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Executive Summary 


The way health services research is conducted is changing, as the health care delivery 
system evolves to be more integrated vertically through hospital and insurance company 
integrated models and horizontally via accountable care organizations (ACOs). As quality 
improvement (QI) teams address clinical problems, research teams study implementation, 
and awareness for prompt assessment and actionable adaptations accrues, the methods for 
pursuing rapid-cycle research are emerging. This document explores the breadth of 
methodologies conventionally associated with QI in primary care settings, discusses the 
appropriateness of such strategies across evaluation objectives, and provides practical 
discussions about tools to accelerate the assessment-to-adoption cycle. 


Purpose: This document is designed as a practical guide to the uses of and methods for 
conducting rapid-cycle research. It was developed by the Agency for Healthcare Research 
and Quality (AHRQ) Practice-Based Research Network (PBRN) Resource Center to: 


e address a demand of researchers and implementers, including the primary care PBRN 
community to describe the process of identifying opportunities to pursue rapid-cycle 
research, and 


e provide in-depth exploration of some methodologies of interest. 


Audience: Research teams conducting health services research in primary care and other 
health settings; quality improvement staff in health care settings, and physicians and other 
providers involved in practice-based research networks. 


Definition of Rapid-Cycle Research: The working definition of rapid-cycle research for this 
document is a process by which practical problems are identified and addressed using 
analysis methods that are incremental and contextually informed. Six phases by which to 
conduct this rapid iterative process are outlined and described with the emphasis on the 
preparation phases of designing a study. 


The phases in the process are: 


1. Preparation: The preparation phase involves 
identifying partner organizations and the 
individuals within those partner organizations 
who will champion the study. It also means 
identifying potential opponents within the 
system and understanding their perspectives. 


2. Problem Exploration: The problem 
exploration phase involves understanding 
patient/provider problems that are important to 
solve. Four tools to help develop deeper 
understanding of the problems that need 
solving or questions that need answering are 
provided: 1) the walk-through, 2) the nominal 
group technique, 3) the critical incident technique, and 4) dialogue. 
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3. Knowledge Exploration: The knowledge exploration phase explores the problem 
from different perspectives. The four steps of knowledge exploration are: 1) 
characterize the problem in the most general way possible, 2) identify other 
industries that face the same problem as it is characterized in a general way, 3) 
identify organizations in those industries that have best addressed the problem, and 
4) identify the processes or activities that differentiate the best organizations from 
others by how they addressed the problem and determine what makes them so 
much better at this than their peers. 


4. Solution Development: The solution development phase identifies the simplest 
possible solutions that can be applied. Applying an engineering process called “ideal 
design” is recommended to devise the simplest, least invasive, and most scalable 
solution to the problem. 


5. Solution Testing: The solution testing phase is the process for determining if the 
identified solution works. It can be done in a rapid-cycle pilot testing method as in QI 
or may be conducted with more traditional research methods. The testing method 
used is determined by the question that is asked and the uses of the answer. This 
section also highlights some research methods being used in practice-based 
research settings that may accelerate the research-to-implementation cycle. 


6. Implementation and Dissemination are the final steps in the rapid-cycle research 
process. We provide several resources to support implementation and 
dissemination. These resources help research and QI teams scale up their efforts 
once their solution is found effective, and address implementation issues providers 
might encounter. 


The appendix contains an annotated bibliography and several resources that research and 
QI teams can use to adopt more rapid-cycle research methods. 
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1.Introduction 


1.1 Purpose 


This document is a practical guide to the uses of and methods for conducting rapid-cycle 
research. It was developed by the Agency for Healthcare Research and Quality (AHRQ) 
Practice-Based Research Network (PBRN) Resource Center to: 


e address a demand of researchers and implementers, including primary care PBRN 
community to describe the process of identifying opportunities to pursue rapid-cycle 
research, and 


e provide in-depth exploration of some methodologies of interest. 


This document will help research teams accelerate the research cycle from concept 
development to implementation in practice. Greater depth of discussion is provided on the 
preparation and pre-study phases of the research cycle because, despite being essential to 
accelerating the research process and ensuring that findings from research are adopted, they 
are often underexplored due to pressures to “get started” in the field. The framework of 
phases presented is oriented around the “Secrets of Rapid Research Framework,” based on 
the work of David H. Gustafson.' This framework outlines steps in the Ql/research process 
and links these to tools and resources that accelerate these processes. 


1.2 Target Audience 


Research teams conducting health services research in primary care and other health 
settings, quality improvement staff in health care settings, physicians, and other providers 
involved in practice-based research networks. 





1.3 Objectives 

1. Define rapid-cycle research. 

2. Describe a process for conducting rapid-cycle research. 

3. Use tools for conducting rapid-cycle research. 

4. Provide examples of where rapid-cycle research has been used. 
1.4 Why Rapid-Cycle Research Is Important 


Rapid-cycle research is important to ambulatory care research settings such as PBRNs 
because it results in better care faster.” 
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PBRNs were designed in the late 1970s to address the concern that research was not used 
in practice. They engaged individual and small groups of clinicians in practical research; but 
today’s practices have changed. Clinicians now are often organized as large groups under 
hospital or vertically integrated payer/provider systems. In these systems, administrators— 
not just clinicians—must approve participation in studies. Also, clinicians in large systems 
must know and manage the outcomes for populations, not just individual patients. 
Productivity expectations limit their time to design research projects, collect data, and 
perform other research-related activities. Efficiency has become as important as 
effectiveness in practices. Clinicians may avoid participating in research that does not fit into 
routine workflows or address problems that they perceive. 


PBRN Membership Demographics eats, 
i a wen 
Across the 173 Registered PEN Sos 7 : pbrn ° 
e 29,325 member practices or clinics Se “= 

o Onaverage, each PBRN has 170 member practices. RCH WE 


o The median number of practices per PBRN is 43. 
e PCMH Designation 

o 94PBRNSs have practices with PCMH designation. 

o 78 PBRNs have practices pursuing PCMH designation. 
e 153,629 clinicians across practices 


As our health care delivery system changes, roles and responsibilities of different players 
evolve. The widespread use of QI staff and processes blurs the distinction between QI and 
research methods. While quality improvement staff focus most of their efforts on rapid-cycle 
QI using proven methods (e.g., Plan-Do-Check-Act [PDCA] cycles), they may also identify 
issues that need a clinical instead of a process response. They may use research methods 
to answer their questions, or blend QI and research methods. When quality monitoring 
identifies research questions, research can be designed to address practical problems. 
Research designed to answer practical questions raised in clinical practice may be 
conducted differently from research 
using the current grant-funded 
model (i.e., research addressing a 
single question, taking years to 
plan, fund, study, and publish). 


Another change that has occurred in 
the past 10 years is the availability 
of a vast quantity of data via 
electronic health records (EHRs), 
patient recorded data from activity 
trackers and other monitors, and 
self-management support tools. 
These data sources provide an 
opportunity for research to be quickly conducted at a population level. 
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In summary, changes in system design, staffing, and data collection and management 
require us to rethink how we conduct research in primary care settings. This guide gives 
researchers and primary care staff tools to quickly conduct practical research within the 
opportunities and constraints that exist in their current delivery structure. 
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2.What Is Rapid-Cycle Research? 


In their 2014 paper, C.J. Peek and his colleagues identified five criteria for research (the 
“Five Rs”) to address the problem of making research relevant in a rapidly changing 
environment. One of their Five Rs is rapid and 


Five Rs of Health Care Delivery recursive research design. This design 
Research accelerates the research-to-practice process by 
1. Is relevant to stakeholders using real-time clinical data, engaging 

2. Is rapid and recursive in application stakeholders to identify research questions, 

3. Redefines rigor ddeciani fudias thal ie like th 
4. Reports on resources required SE Ge TING Stueies Viet Cal Operate ine ite 
5. Is replicable PDCA cycles used in QI processes. The Five R 


model has been promoted as a general 
standard in primary care health services 
research.’ 


The literature defines rapid-cycle improvement 
as a process that can be done in a brief period 
(i.e., a month or less) and includes multiple 
cycles of small changes to address a problem. 
In the early days of implementing rapid-cycle 
improvement in health care, cycle times of 90 or 
fewer days were described as rapid. Experience 
using and adapting the methods reduced the 
perception of how much time a QI process 
should take. Elements that were changed in 
process improvement methods to speed up the 
process are also relevant for accelerating research today. These elements include solving 
key problems; minimizing meeting time; being selective about measures; and considering the 
ability of each tested solution to be scaled up, disseminated, or continued outside of the 
experimental time frame and conditions before it is tested.° 





There is a growing body of literature on rapid-cycle evaluation. Rapid-cycle evaluation 
applies program evaluation methods early and often in a pilot program’s or policy's 
implementation process. Then, rapid-cycle improvement methods are used to change the 
program or policy to address issues raised by the early findings.® Because researchers 
collect and analyze data early in the process rather than waiting until the end of a pilot or 
study, the intervention can be adapted, the change tested, and iterative improvements made. 
While this method is best known for how the Centers for Medicare & Medicaid Services 
(CMS) deploy it in their study of health care finance and delivery system models, it can also 
be applied to clinical research. Examples of this include applying rapid-cycle evaluation to 
assess technologies’® and care improvement processes. 


Rapid-cycle research is still a relatively new term. As in the early days of rapid-cycle 
improvement, the experts consulted clearly understand which processes take too long, but 
they do not necessarily have a clear definition of rapid-cycle research. However, there is 
great interest in shortening the research-to-adoption cycle and in applying the knowledge 
gained from rapid-cycle improvement processes to the research arena. The definition of rapid 


I pg. 4 





o © | 


Preparation Problem Exploration Knowledge Exploration Solution Development Solution Testing 





AHRQ PBRN Using Rapid-Cycle Research to Reach Goals: Awareness, Assessment, Adaptation, Acceleration 





and how to include cyclical processes in research design are points still being debated and 
tested. 


A Working Definition of Rapid-Cycle Research 


This document’s working definition of rapid-cycle research is: a process by which practical 
problems are identified and addressed using analysis methods that are incremental and 
contextually informed. Six phases used to conduct this rapid iterative process are outlined in 
a graphic depiction of the Framework for Rapid-Cycle Research in Exhibit 1.1. A bicycle gear 
highlighting the six phases depicts rapid-cycle research’s cyclical nature. At any point in the 
research process, methods from earlier phases may be incorporated to make adaptations 
based on new information. The contextual awareness examined in each phase of this 
process enables the mechanisms of change to be identified efficiently, thus rapidly 
accelerating the assessment, adaptation, and adoption of solutions. 
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Figure 1.1. A Graphic Depiction of the Framework for Rapid-Cycle Research 
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Solution Development 
Solution Testing 


Implementation 
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2.1. Framework for Rapid-Cycle Research 


We based our framework for rapid-cycle research on the work of David H. Gustafson (see 
Exhibit 2.1.) in process improvement and research.’ The framework outlines six phases of 
the Ql/research process, from conceptualization through implementation, and provides a set 
of tools that improve each phase’s efficiency and effectiveness. By thoughtfully executing 
each phase of the process, researchers may accelerate both the research and adoption 
timeframes. Although this framework is organized from the first steps of preparing for the 
research process through the implementation of research findings, some of its tools and 
resources can be used in more than one phase. 


Figure 2.1. Dr. Gustafson’s “Secrets of Rapid Research” Framework 

















: Influential change fe 
Preparation < | Committed Team ieacisk Simplicity 
Problem Nominal Group i 
Exploration @ Technique Walk-Through Critical Incident 











Plan Do Check Act* 





Implementation 
: + ati Address 
and Dissemination . 
Fe. environmental and Monitor and Alert Stories 


organizational factors 





* The Plan Do Check Act Model can be used during the Knowledge Exploration, Solution 
Development, and Solution Testing phases. 
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3.Applying the Phases of Rapid-Cycle Research 


3.1. Preparation Phase 


The preparation phase involves 

identifying partner organizations and the 

individuals within those partner 

organizations who will champion the DART Net \nstitute 

study. This phase also involves Informing Practice Improving Care 

identifying potential opponents within the — There are some tools for a project’s Preparation 

system and understanding their Phase. The DARTNet Institute’s “Toolkit for 

perspectives. Developing and Conducting Multisite Clinical Trials in 
_ Practice Based Research Networks” (see 

BESearenironi BRO :s Integrated http://www.dartnet.info/clinicaltrialsPBRNtoolkit.htm)'° 

delivery system research network includes information on partnering with clinical 


: : erica feplaced wih Ine: ACTION practices, budgeting, and planning for dissemination. 


http://www. ahrg.gov/research/findings/fa 

ctsheets/translating/action/index.html) showed the value of a carefully planned preparation 
phase in rapid-cycle research. In evaluating the IDSRN model, four factors were identified as 
facilitating successful research-to-practice efforts: 





1. having a research team employed by the health system or a champion in the health 
system when the research team was external, 


2. the study’s responding to delivery system needs, 
3. ongoing funding, and 


4. creating tools (e.g., decision support tools) to 
facilitate adoption. 


Factors that hindered success were multiple 
demands that drew the clinical team away from the 
research process and failing to address the correct 
audience." 


Our own experience suggests that, in addition to the 
factors identified in the Gold'' study, having a strong 
leader for the research team and having clear 
timeframes and objectives improve the probability of 
moving research findings into practice. 





These concerns must be addressed in the preparation phase before a study is begun. We 
cannot overemphasize that simplifying the study process during the planning phase 
increases the likelihood of developing strategies that will provide sustained improvement in 
care. 
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3.2 Problem Exploration Phase 


Identifying key problems for stakeholders (e.g., clinicians, funders, and especially patients) 
engages them in the problem-solving process. It accelerates recruitment of study participants 
and adoption of study results. Figure 3.1. below reflects the importance of engaging clinicians 
in rapid-cycle research. 


Figure 3.1. Diagram of engaging clinicians in rapid-cycle research’ 





Practice-based Research 











Four different techniques to identify problems worth solving that also engage stakeholders 
from the earliest phases of designing a research or QI project are listed and described below. 
Each has its own purpose, and may be used alone or in combination with the others. 


1. The Walk-Through—Purpose: understand the experience of care from the “customer” 
perspective. 


2. Nominal Group Technique—Purpose: engage the research/care provision team in 
identifying possible causes of problems. 


3. Critical Incident Technique—Purpose: help the “customer” describe an experience so 
researchers and care providers can better understand needs and strengths within the 
context of the experience of using the health care system. 


4. Dialogue—Purpose: develop a common understanding of a problem based on 
sharing different perspectives. . 


3.2.1. The Walk-Through 

The walk-through tool has been used in a number of 
industries including manufacturing, '* restaurants, '? and 
health care.'* It is a process of “walking” through the 
experience in which problems have been identified. For 
example, in manufacturing, managers walk through the 
factory to better understand how bottlenecks and defects It is important to engage patients in 
occur. In restaurants, the manager approaches the the problem exploration process. 





' Reprinted with permission from Paul Nutting, MD, MSPH, Director of Research, Center for Research Strategies. 
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restaurant as though she were a customer and walks through the whole experience to 
identify issues from signage, service, meal preparation, etc., from a customer perspective. 
The idea in each case is to better understand the process and how it works or does not work. 
Ultimately, this identifies what can be done to create a better product or improve the 
customer experience or increase efficiency and reduce mistakes. In health care, walk- 
throughs have been used to identify access and utilization issues.'° The walk-through can 
also be used to identify research problems that address patient and provider concerns. 


@, 

How to do a Walk-Through: ? 
1. Identify the process one wants to walk-through. Specify the beginning and 2» 
ending points. { 


2. Let staff know one will be conducting a walk-through and tell them it is to 
understand the process and “customer” experience, not to find out what 


ay: 


they are doing wrong. Note that during the process, the staff will be asked wf 
to suggest changes that will make things better for them and their ‘ 
customers. The walk-through is a joint effort (researcher and staff). 

3. Develop a character based on the process one is trying to understand: for @o, 
example, “I am a 50-year-old female going for her first chemotherapy * 


appointment for stage three breast cancer,” or “| am the partner of a 50- 
year-old female with breast cancer’ or “I am a nurse with 20 years of 
experience trying to implement a new electronic health record population 
management tool.” 

4. Conduct the walk-through by completing each step of the process as 
though one were the character created. One should ask to be treated as if 
they were that character and act exactly as they would for that character. 
Completely fill out all forms and go through every step of the process. Think 
about what the experience feels like from that character’s perspective. Ask 
staff questions about what they are doing and why. Ask the questions in an 
open way to assure them that one wants to understand how the process 
works, not to catch them doing things wrong or making mistakes. Take 
notes. Since two sets of eyes are usually better than one, ask a second 
person to play the role of family member or another relevant role and take 


notes. 

5. Discuss the experience with one’s change/research team. What was 
learned that might be a problem that is worth solving or what is a question 
that is worth asking? What strengths were identified that might be used as 
part of the solution? 


Done properly, the walk-through gives management or the research team a patient’s view of 
a specific treatment and the care delivery system that provides it. Patient concerns may be 
process-oriented, but they may also be treatment-specific. This tool can also be used to 
better understand other stakeholder perspectives by, for example, standing in the shoes of a 
clinician or a nurse to identify why a certain process is not being adopted and deciding how it 
might be changed to be better incorporated into the normal work flow (or how the work flow 
needs to change to adopt the practice). 
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3.2.2 Nominal Group Technique 

The nominal group technique (NGT) is a proven tool for identifying problems. Over 40 years 
ago, Van de Ven"* proposed it as a way to identify worthwhile health research questions. A 
relatively simple process that engages members of health care delivery teams, it can help 
identify both the causes of problems and their potential solutions. Van de Ven suggests a 
researcher use the NGT to clearly define the dimensions of the problem so that data 
collection is relevant, valid and reliable. The nominal group process has five steps’” (see 
Appendix D for a detailed description). The strength of NGT are that it reduces the risk of 
one person (or a few people) dominating a discussion.'® It does so by limiting and controlling 
conversation. For instance, prior to any discussion, the facilitator asks that each person ina 
group silently think through the task and generate his/her own list of ideas. Rather than 
having an open discussion, each person is asked to give only one idea and then move to the 
next person and continue around the table until all ideas are out, allowing each person an 
equal opportunity to contribute. 


The Nominal Group Technique’ has five steps: 


Step 1: Preparation 
Step 2: Silent idea-generation 


1. 
2 
3. Step 3: Round-robin recording of ideas 
4. Step 4: Serial discussion of ideas 

5 


Step 5: Preliminary voting 
a. Rating method 
b. Ranking method 


3.2.3 Critical Incident Technique 

The critical incident technique (CIT) was developed in the 1940s for psychological studies.°° 
The CIT originated in studies to better understand decisionmaking and to develop 
assessment tools that could objectively identify capabilities (e.g., piloting and leadership) in 
the military. It was later used to identify the criteria for assessing the ability to perform a 
variety of roles. 


CIT has been used to understand patient experiences in health care,*' to view how health 
care teams function,” and to identify and reduce medical errors.°° 


Critical Incident Technique has five stages: 


1. Identify the aim and research question. 

2. Plan (e.g., identify incidents to examine and the means to access them). 

3. Collect data by direct observation or in interviews. The goal is usually to better understand 
what a person is thinking or experiencing during an incident. Thus, surveys may work, but 
they often do not collect data at the level of depth desired. 

4. Identify themes by analyzing data using qualitative research methods. 

5. Interpret and report results. 
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The CIT’s purpose is to get someone to talk about an experience in detail. Figure 3.2. 
outlines the distribution of PBRN communication modalities as reported in the 2015 PBRN 
Registration Responses. 


Figure 3.2. Distribution of PBRN Communication Modalities (N=173) 
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Ultimately, that data will help the researcher guess the person’s needs. People often do not 
know what their needs are, but they can describe an experience and how they felt about it, 
and researchers can work effectively from there. Below is an example. 
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Example of Critical Incident Technique to Understand Needs of a Breast 
Cancer Patient: I 
Instead of asking a breast cancer patient what she needs, Dr. Gustafson and I 
colleagues asked: “What is it like to be a woman with breast cancer?” There | 
are several important points to consider when phrasing the question. First, | 
even for health care providers whose interest is improving their cancer service, | 
it is important to get a broader picture of their customer. Asking patients what | 
it is like to be a patient in a clinic may help improve a clinic’s internal I 
processes, but it limits the scope of solution finding. Three types of service 1 
qualities exist: expected, desired, and unexpected. Expected qualities address 
the things that are assumed to be present. United Airlines does not advertise 
using the slogan, "We don't crash,” since customers would not fly with them if I 
they expected United’s planes to crash. Similarly, patients expect that I 
practices will have skilled clinicians. If someone experiences a crash or I 
encounters a medical error, the cost can be enormous. Desired qualities of | 
service, in contrast, are ones customers know to hope for, but realize they may | 
not get. Examples of desired qualities include easy parking, minimal delays, I 
nice people, and tasty food. Unexpected qualities are different: patients do not I 
know that they need these in their health service experience, but they are often 1 
overjoyed when they experience them. For providers to deliver unexpected 
qualities, they must know their patients. Asking what it is like to be a patient 
within a particular practice may help improve expected and desired qualities, I 
but that question does not give a broad enough picture of a provider’s I 
customer to address unexpected qualities. To get at unexpected qualities, I 
providers and researchers must know what it is like to be that woman with | 
breast cancer, including being aware of critical incidents that have deeply | 
affected her life both within and outside of the health care delivery system. I 
The point is to probe to really understand the experience of being the person I 
with that condition. Dr. Gustafson and colleagues probed about what the 1 
experience was like: what she felt, wanted, hoped for, and worried about to ; 
better understand how to deliver the unexpected qualities that would improve 

her condition. This deeper understanding was important as Dr. Gustafson and I 
colleagues developed information communication technologies for patients I 
with breast cancer in the early days of personal computers.” I 


ee ll 


4 
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3.2.4 Dialogue 

Dialogue is a uniquely powerful methodology for achieving shared understanding and 
common purpose. As a formal method for group communication, Dialogue emerged from 
Bohm,” a theoretical physicist, who observed that many of the world’s intractable social 
problems stemmed from distorted thinking, fixed beliefs, and a lack of shared understanding 
between people with fundamentally different life experiences. The techniques and attitudes 
that characterize formal Dialogue have evolved and are well described in books by Isaacs*® 
and Ridings.”’ The methods of formal Dialogue have been successfully used to improve the 
working environment for health care workers,” resolving longstanding and bitter disputes 
between management and labor,” and facilitating deeper appreciation of unconscious racial 
bias among individuals with diverse experiences of privilege, power, and discrimination.°° 
Dialogue requires suspension of judgment, respect for differences of opinion, intentional 
listening, and giving voice to different perspectives. 
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Many researchers know the value of open 
conversation and careful listening. Working 
productively with research or QI colleagues or 
stakeholders from diverse educational, ethnic, racial, 
or socioeconomic backgrounds requires openness, 
patience, and the desire to listen carefully. In 
particular, researchers who work on multi- 












entrenched differences 
among stakeholders 
and/or constituents 
seeking to generate 
research or to rapidly 
cause improvement. 


a lack of shared , . 
understanding and unconscious bias 


common purpose, 










Dialogue may be useful 
when the deployment of 
rapid-cycle research or 

rapid-cycle improvement 
is hampered by: 





disciplinary or transdisciplinary teams; use multi-method approaches (e.g., qualitative and 
quantitative methodologies); or engage patients, communities, or stakeholders to develop 
research questions and hypotheses, collect and interpret data, and disseminate and 
implement findings must be adept at communicating. One might call this a “dialogic’” 
approach to communication, regardless of the setting, circumstances, or people involved. 
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Dialogue as a method refers to a more specified, complex, and formalized process of 
communicating in groups. Individuals with formal training and experience with Dialogue bring 
a dialogic approach to all conversations, including one-on-one and group meetings. Having a 
dialogic orientation and well-developed dialogic skills leads to very effective communication. 


Dialogue as an explicit method of obtaining a common understanding and learning from the 
collective intelligence of a group promotes a deep understanding. The level of understanding 
achieved with Dialogue is deeper and richer than what individuals can achieve alone or in 
ordinary conversations or discussions. 


3.3. Knowledge Exploration Phase 


Many research projects inadequately explore alternative solutions. Once a problem is 
identified, people often coalesce around a single 
solution without adequately exploring alternatives. 


Exploring alternative solutions may seem to be an 
added step that increases time in a research cycle. 
However, spending time in this phase may increase 
the odds of finding a successful solution and 
significantly reduce time in later phases. 
Researchers may identify more easily tested 
solutions or solutions that are more easily adopted if additional time is spent in this phase of 
a Ql/research project. 





One way to do knowledge exploration is by a stepwise process that begins with the most 
general level of characterization of the problem, and then narrows the focus to identify 
elements that must exist in any successful solution. 
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f Knowledge Exploration-Population Management Characterized as a Hand-off Problem: 





| A population management system identified the problem that patients with diabetes have a 
I lower rate of blood sugar control than meets CMS criteria. After further data analysis, the 
| problem of a low rate of follow-through on referrals to the dietician and endocrinologist was 
| identified. One way to characterize the problem is the failure in making a hand-off. Many I 
industries have hand-offs. Manufacturers have hand-offs on the assembly line, sports teams I 
(e.g., basketball, relay races) have hand-offs. Even airplanes are handed off between air traffic |! 
control towers. Football is an example of an unrelated industry where good hand-offs are I 
I critical to success. What makes a football team have great success with hand-offs? There are _ | 
clear signals for which play is going to be called, and each player knows where to go. Because | 
everyone is in the designated place, the quarterback knows where to throw the ball, or to whom ; 
I he should hand the ball at the right moment. These actions occur smoothly because the I 
I players practice so often. I 
We see similarities in how good hand-offs occur if we take the football teams’ solutions and I 
apply them to hand-offs of airplanes between air traffic control towers. Air traffic controllers I 
I have a manual for conducting hand-offs of planes. They practice the process many times | 
daily. The roles and activities of each air traffic controller are clearly defined in the manual, I 
I and the hand-off is not complete until both air traffic I 
I controllers acknowledge it. , 
How can we use these ideas to solve the problem of l 
patient follow-through with referrals? Patients are I 
not inanimate objects that can be handed off like a | 
I football. Yet, patients benefit significantly from a 1 
| clear process in which each member of the health 
j care team has a defined role and responsibility. I 
, Communication is clear concerning where each l 
team member is in the process, and it is the team | 
members, not the patient, who must ensure the I 
| patient gets appropriate and safe care. These are the essential ingredients of a good hand-off ; 
| that can be used regardless of the industry. 
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3.4 Solution Development Phase 


Innovations that are widely adopted have characteristics that Everett Rogers identified in his 
work with the University Extension Systems.°*' Combining relative advantage, compatibility 
with current processes and beliefs, simplicity of implementation, trialability, and observability 
increases the probability that an innovative solution will be adopted. In other words, these 
are attributes of solutions worth testing. In health care systems, because of the rapid pace of 
change in things beyond clinician control, providers need solutions that decrease their 
workload rather than adding to it. Solutions that meet the criteria for having a higher 
probability of adoption are also easier to test (trialability is one of the criteria). The concept of 
ideal solutions is one tool to develop a solution that meets the criteria listed above. This 
process allows one to address the fundamental dimensions of a solution design: purpose, 
inputs, outputs, sequence of steps, environment, human resources, equipment, data, and 
capability to improve. 


3.4.1. Ideal Systems 

A systems engineering approach to solution development is to think in terms of an ideal 
system. How would the procedure ideally be delivered? How would the process function if 
cost were not a consideration, if there were no waits, and no special effort were needed on 
the customer's or staff's part? This gives researchers a target that may be unrealistic, but 
one’s goal is to see how close they can come to the ideal. When designing an intervention 
with the starting point that the ideal solution would require no effort, time, and costs, one is 
likely to come up with very innovative ideas. Of course they will not be realistic, but moving 
from the ideal to the realistic is the next step. 
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The point is to say: “Here is our customer, and here is what it is like to be that customer.” 
More than anything, what does that customer need that he or she does not currently have? 
Now, assume that anything is possible to meet that need. Cost is not a consideration. 
Limitations of current technology are not a consideration. One can have anything they want 
to meet that customer’s needs. But making the patient's life more difficult is a limitation. 
Creating bureaucratic burden for staff is a limitation. What, in the ideal world, would that look 
like? That ideal world becomes the starting point for thinking about possible solutions to test. 


Example of using Ideal Systems Design: 

In the late 80s Dr. Gustafson and colleagues were trying to help women cope with a diagnosis 
of breast cancer. The team conducted several critical incident interviews to understand their 
needs (e.g., One woman said, “The surgeon told me | had breast cancer. S/he said, ‘You need 
to have either a mastectomy or a lumpectomy with radiation. Here are the advantages and 
disadvantages of each. | can’t tell you what to do, but you should decide within a couple of 
weeks.’ | felt so lonely.”) Dr. Gustafson and colleagues convened a group of seven women 
with breast cancer who were considered to be very innovative thinkers. They were asked, 
using a nominal group, to silently generate descriptions of the ideal help that would meet their 
needs. One woman wanied an 18 wheeler left at her home. It would be filled with all the 
literature on breast cancer translated into a language she could understand. Another had an 
extra bedroom, and she wanted her personal oncologist to move in. Another wanted five 
women just like her (except that they were at different points in their journey—just diagnosed, 
5 years out, etc.) to move in with her so that she could talk to them at any time. Of course, 
none of that was possible. They then convened a group of technological visionaries and asked 
them to come up with a solution that was as close to the ideal as possible. By the mid-1990s, 
hundreds of women were using a computer system (CHESS) that had an instant library (instead 
of an 18 wheeler), an anonymous ask-the-expert (instead of a personal oncologist), and a 
bulletin board discussion group (instead of the women moving in), and there was a decision 
analysis program to help them think through the type of surgery they might have.” 
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3.4.2. Using Change Concepts 
Another way to simplify solution development without settling on one solution too rapidly is to 
consider the types of solutions that have been developed in other studies or to solve other 
problems. Similar to using the steps in the knowledge exploration phase, this process 
identifies the possible solutions in high-level conceptual categories that may be applied to 
multiple types of problems, then considers how 
they might work to address the current problem. 
After identifying a category of solutions, the 
concept can be refined to fit the specific problem 
being solved. 


For example, Langley et al®* identified nine 


categories of change concepts (Such as improve 
workflow) with a total of 72 specific ways (e.g., 
minimize hand-offs) to think about how a problem 
could be solved. Providing a selection of ways to 
categorize a problem (e.g., this could be a work 
flow problem) and solutions (e.g., what if we 
reduced the number of hand-offs?) may help QI and study team members rethink a problem 
and develop more creative solutions. 








3.5 Solution Testing Phase 


Campbell and Stanley,” in their classic 71-page text on research designs, focus on the 
importance of deciding what kinds of things are likely to interfere with one’s ability to 
confidently state whether the intervention being tested is really having an effect. 


Below we have listed 10 threats to internal validity that impact one’s ability to state that the 
intervention is what is causing the effect. The first seven are important if one is not trying to 
generalize beyond the people being studied. The last three are also critical if one hopes to 
generalize. One should take a few minutes to sit back and think about the study: the 
interventions, the subjects, the measures, the environment. One should go over the 10 
threats and circle how worried one should be about each. 


1-None Some Very. History: the world will change around you (e.g., maybe a change in 


leadership?). 


Solution Development Solution Testing 
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2- None 


3- None 


4- None 


5- None 


6- None 


7- None 


8- None 


9- None 


10- None 


Some 


Some 


Some 


Some 


Some 


Some 


Some 


Some 


Some 


Very. Maturation: the subjects will change (e.g., age out of the problem, 
leave town, become more infirm). 


Very. Testing: just by completing the pretest, the subjects will have 
learned things that will affect their performance on the posttests. 


Very. Instrumentation: the measurement methods may change (e.g., a 
new format; an old measure is replaced by a “better” but different 
one). 


Very. Regression: the study group is selected because group members 
recently performed particularly well or poorly (e.g., | might have one 
good day at golf, but the next time | will not be so lucky). 


Very. Biases in selection: |t you are comparing one group to another, are 
there reasons to believe that one of those groups is different from the 
other to start with? 


Very. Experimental mortality: |f you are comparing two groups, is there 
reason to believe that subjects from one group will be more likely to 
drop out of the study? 


Very. Selection-intervention interaction: lf you are comparing two 
groups, is one group more likely to respond to the intervention than 
the other (e.g., if one set of subjects comes from football players and 
another from horse thieves, and the intervention is teaching them to 
ride a horse, the horse thieves are more likely to be experienced in 
riding in the first place)? 


Very. Reacting to the testing: the pretest will change the patient in ways 
that makes them different from real-world people who would not be 
taking that test (e.g., suppose the pretest was time to complete an 
obstacle course. Just taking the pretest may make these people feel 
that they need to improve their conditioning. Even if the intervention 
itself has no effect at all, the subject may improve anyway.). 


Very. Reacting to being in an experiment: Could the way the 
experiment is being conducted be so different from the real world 
that results will not be the same in the real world? (e.g., the 
Hawthorn studies measured whether productivity improved when 
lighting improved. It did. Once the lighting was at the highest level, 
they measured what happened when lighting was reduced. 
Productivity improved further. It was not the lighting. It was that they 
were being studied). 


Once the biggest threats are identified, one can select a design that will protect from those 
threats. Campbell and Stanley™ provide tables that allow 16 different study designs to be 
compared in terms of their ability to protect against the various threats. For instance, they 
show that the classic randomized controlled trial (RCT) is great as long as one does not try to 
generalize from the results. Whereas separate sample pretest-posttest designs are great for 
generalizing, but are not nearly as good for the first seven criteria. The nature of the research 
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aim—whether explanatory or pragmatic—should also be considered. Figure 3.5. “Flow of 
Analytical Methods’ is a whimsical illustration of the choices in research objectives and flow 
of resources that should be activated based on data resources, research objectives, and 
anticipated application of findings. The selection of the appropriate combination of qualitative 
and quantitative analytical methods to apply is determined based on the objectives of the 
study and contextual factors that explain the intent and implementation of the intervention. An 
appropriate combination of qualitative and quantitative methods also provides data to support 
replicability and translation of study findings to other situations and settings. 


Figure 3.5. Flow of Analytical Methods 





3.5.1 RE-AIM as a framework 

Once potential solutions and testing strategies 
have been identified, it is worth using RE-AIM®° 
as a framework for considering how well one’s 
proposed solution addresses all the issues that 
might hinder wide adoption. RE-AIM stands for 
reach, efficacy, adoption, implementation, and 
maintenance. The model was developed as a 
tool for assessing and comparing interventions in 
their likelinood to achieve a sustainable 
improvement. Glasgow recommends that 
researchers attend to issues of external validity 
equally with internal validity and provides a list of 
things to consider. The RE-AIM Web site 
(http://www.re-aim.hnfe.vt.edu/index.html) has 
turned this list into an online tool and checklist for 
evaluating a study plan within the RE-AIM framework. For a presentation on using the RE- 
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AIM framework in assessing the level of pragmatism in a study, see the AHRQ PBRN 
Resource Center Webinar How pragmatic is it? Lessons learned using PRECIS and RE-AIM 
for determining pragmatic characteristics of research at http://pbrn.ahrg.gov/events/how- 
pragmatic-it-lessons-learned-using-precis-and-re-aim-determining-pragmatic. 


Not all questions need to be answered using an RCT. Other methods may be adequate to 
answer a particular question and may be executed more rapidly than a typical RCT. 
Campbell and Stanley explain that either more data or more assumptions are necessary to 
assert causation via quasi-experimental designs. They recommend more data over more 
assumptions. The large quantities of data that are available now via electronic health records 
(EHRs)—and that will soon be widely available via patient self-monitoring tools (e.g., 
wearable devices) and through communication via mobile applications—make it possible to 
have the quantity and type of data needed to assess the impact of different interventions. 
Some PBRNs are using data from EHRs to conduct multisite studies designed to address 
care delivery quality. These large-scale projects use data from the EHR to identify common 
practice issues, test practice improvements, and assess the targeted interventions. For 
example, the Wisconsin Research Education Network (WREN) combined data from EHRs 
and community data regarding population density and air quality to predict asthma control 
and identify targets for improving asthma control with predictive analytics.*” The Canadian 
Primary Care Sentinel Surveillance Network used EHR data to identify variation in practice 
and outcomes, develop and test clinical algorithms to address that variation, and assess 
sustainability of improved practices.*® 


In a presentation at the North American Primary Care Research Group (NAPCRG) on July 1, 
2014, Wilson Pace*’ described the advantages of “big data” to practice-based research (the 
presentation slides are available here: (PDF — 900 KB). 


e The ability to randomize using baseline practice performance and characteristics. 
e The ability to balance multiple criteria at once, even with small groups of practices. 


e The presence of historical data allows tailoring of academic detailing to each site if 
needed. 


e Measurement of a broad spectrum of outcomes, cost analysis, and intervention 
effectiveness can be completed without asking clinical staff to do more than their usual 
data collection. 


e Providing a feedback loop to practitioners creates an adaptive health care system. 


However, researchers who have begun using EHR data have raised issues about data 
quality. They caution that EHR data may need to be carefully cleaned before it is useable in 
research or QI projects. A recorded Webinar from DartNet on using clinical records for 
research is available at 
https://altarum.adobeconnect.com/_a758956138/enrich120612archive/. This presentation 
covers issues of data quality, pluses and minuses of EHR data and examples of studies that 
successfully used EHR data. 





3.5.2 Research-Quality Data 
The concept of data quality includes accuracy (validity), precision, completeness, and 
consistency and appropriateness. Data quality problems can occur from lack of 
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standardization of the data collection processes, poor design of questions and definitions, 
lack of understanding by patients, difficult data collection forms, poor handwriting, 
measurement equipment errors, improper rounding of measures, miscalculations, human 
error, or database errors. Data quality control and improvement are the processes of 
detecting, fixing, and preventing poor data quality. With electronic data systems, most of 
these processes can be automated, including duplicate entry, data audits, comparisons to 
reference data, checks for boundaries and reasonableness, and other matches to pre- 
programed expectations. 


Regardless of the research design used, considering data quality through all phases of the 
research project is important. Data quality control is part of ensuring the integrity of the 
research results. Rapid-cycle research designs have some advantages over traditional 
research relative to data quality. Using rapid-cycle methods where data is reviewed early and 
often deals with data quality issues early, making it easier to solve the current problems and 
to prevent other problems from recurring. 


QI and research projects can take advantage of a registry where data describing both the 
patient and the care processes are entered in real time (or close to real time) during a patient 
visit. Sometimes this can be done directly from the EHR. Exception reports can thus be 
created in real time and actions taken to mitigate data quality problems even before the visit 
is complete. Monthly reports may include data quality statistics for review and action. These 
actions create quality data that are useful for research purposes. 


3.5.3 Rapid-Cycle Testing Designs 

Use of any of the testing strategies described below to accelerate the research-to-practice 
cycle depends on two factors. The first factor is that the study design is appropriate to the 
question being asked or the problem being addressed and the use to which one will put the 
answer. 


The second factor is a focus on the first four steps of the rapid-cycle testing process. Testing 
methods alone will not accelerate the research-to-adoption cycle. Addressing problems that 
matter with solutions that work for patients and practitioners is more important than using 
new models of statistical analysis or study design. Some of these methods may actually 
extend the testing phase, but will provide more information than a standard RCT. Some may 
take as long as an RCT, but refine the intervention during the study period so that multiple 
cycles of RCTs are unnecessary. The list is not meant to be exhaustive or prescriptive, but 
provides a list of examples of research methods that have been employed in pragmatic 
studies by the experts consulted in development of this document. Another resource that 
describes study designs in pragmatic trials is available on AHRQ’s Patient-Centered Medical 


Home (PCMH) Resource Center at http://ocmh.ahrg.gov/page/evidence-and-evaluation. 


1. Time Series Studies 


Repeated measures are the cornerstone of process improvement efforts where measures 
are taken before a change and then again after a change to see if there is a difference. 
Time series studies using frequent measures are more feasible now that electronic 
means of capturing data can reduce both patient and provider burden. 
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Statistical process control (SPC) is an engineering method that uses repeated 
measures to tease out normal variation from real improvement.” It was originally created 
by Shewhart™' to assess manufacturing processes. The idea in a research context is to 
understand the range of normal variation and to see how an intervention affects the 
pattern. 





SPC has been used as a tool to test whether a change really had an impact in a rapid- 
cycle study to reduce methicillin-resistant Staphylococcus aureus infection in hospitals,” 
improve glucose control,** and reduce the number of inappropriately scheduled births. 
All of these studies used SPC to measure improvement within sites that were 
implementing rapid-cycle testing of practice changes. 


Diaz* notes that using SPC is preferred to conducting a RCT under some conditions 
because it may be faster and cheaper, and, using the example of the change in time to 
death as the outcome for the rabies vaccine, demonstrates statistical significance with a 
very small sample. The Institute for Health Improvement offers an online course on the 
use of SPC in health care (see 


http://www. ihi.org/education/WebTraining/OnDemand/Run_ControlCharts/Pages/default. 
aspx). 


Single case experimental design (SCED) is an example of an experimental method 
that has been improved by the availability of electronic data collection tools.** In single 
case, or n-of-one studies, individuals act as their own control. The individual could be a 
single person, practice, or even community that is under study. In this study design, one 
or more interventions are given to a single “case” over a period of time, and the difference 
in response for that single case is observed. The results from a group of single cases 
can be aggregated to test hypotheses about the effect of interventions. Single case trials 
have been used to better describe variation in response to treatments and to understand 
the change process.*° For this reason, they have been recommended as a tool to study 
behavioral interventions and assess improvement in chronic disease.*’ There are many 
variations on the basic pre-post (AB) design that have been developed to address some 
of the potential threats to validity of this type of design. AHRQ has developed a guidance 
document on conducting n-of-one trials with checklists to guide researchers who are 
considering this methodology; Design and Implementation of N-of-One Trials: A User’s 


Guide is accessible here: (PDF — 900 KB). 
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In community-based research, multiple baseline studies, a variation of single case 
design, have implemented an intervention at different points in time in different 
communities to reduce the possibility the outcome includes secular trends or bias due to 
timing of data collection.*® These studies use a basic single case pre-post (AB) design, 
but introduce the intervention to each individual or site at a different point in time and 
aggregate the results of the individual outcomes to better generalize to a broader 
population. 


Point-of-Care Randomized Controlled Trials (RCT) 


Point-of-care RCTs use randomization to address selection bias concerns, but uses a 
rapid-cycle process more like that of a quality improvement process to compare existing 
treatments in usual care settings. This type of study uses existing data collection tools, 
existing treatment protocols, and has few exclusion criteria to minimize the burden on 
patients and providers. Point-of-care RCTs were first described by Fiore et al”? in their 
work to improve care within a group of Veterans Administration hospitals. By 
incorporating randomization of patients to specific treatments into the QI process, they 
developed a method to conduct rapid-cycle research that includes the strength of 
randomization with the pace of QI. This method of solution testing is also being used to 
design a trial to assess which medication is best for treating neonatal abstinence 
syndrome.°° 


| elieleattrlietalartertestietitentintatantentientatantentatatantenteatntatenteataten 


An example of point-of-care RCT: 

One example (http://www.research.va.gov/services/csrd/ 
first test of the model in a study conducted by eight U.S. Department of Veterans Affairs 
(VA) hospitals in New England. When a patient entered the hospital needing insulin, the 
EHR presented the physician with three choices of how to dose the medication. The first 
was no preference, the second and third were traditional protocols of either weight-based 
or sliding scale dosing. The no-preference option led to the offer to participate in the 
study comparing weight-based and sliding scale dosing protocols. If the doctor had no 
preference and the patient consented, s/he was randomized to one of the two protocols. 
This particular study used adaptive randomization hinged to a Bayesian analysis model 
that increased the percentage of people randomized to the study arm with the best 
outcomes until it was randomizing 99% of participants into the preferred arm. By the end 
of the study, they had not only demonstrated which protocol was best, but had also 
moved most of their practitioners to using that protocol via the study randomization 
process. 





3. Factorial Design 


Factorial designs compare the effect of several interventions at the same time. Factorial 
designs are efficient because they can use a sample size similar to that used for an RCT 
that compares only two interventions. Factorial designs are particularly useful in 
assessing what the individual effect of different parts of a complex intervention is or to 
understand how the different components interact with one another. A good example of 
how to plan a factorial study is available at Penn State’s Methodology Center (see 


http://methodology.psu.edu/ra/most/factorial). 
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A variation on factorial design that PBRNs have used is the efficient orthogonal design.*' 
In this study design, each practice is assigned a unique combination of the components 
under study. Each practice is randomly assigned one of the combinations of 
interventions or components. A study or QI team that wishes to use this method must 
carefully select the combinations of components to be tested because as a fractional 
factorial design, efficient orthogonal design is prone to confounding by unmeasured 
interaction effects. 


Factorial designs and their variations may be used to identify the most likely to succeed 
components of a complex intervention that may later be tested in a randomized trial such 
as the Multiphase Optimization Strategy (MOST),°”°? or they may stand alone as a way of 
conducting comparative effectiveness studies across multiple intervention strategies or 
implementation models” or best combinations of interventions that have been previously 
demonstrated to be efficacious.°° 


4. Stepped Wedge Design 


Stepped wedge designs may actually take 
longer than RCTs where the comparison 
group and the intervention group are observed 
at the same time. However, there are 
situations for which the stepped wedge design 
is more efficient, more ethical, and may 
decrease implementation time, thereby 
decreasing the overall research-to-practice 
cycle time.°© Stepped wedge designs are a 
form of cluster randomized trials where all 
clusters eventually receive the intervention, 
but the intervention is introduced at different 
times for different clusters. Stepped wedge studies can be designed as repeated 
measure cross sectional studies or as cohort studies. They work best when the outcome 
measured is a short term outcome or is an interim measure so that patients do not need 
to be followed for long periods of time. They are useful when there are only a small 
number of clusters that are available to participate in a study. Stepped wedge designs 
are being used to study knowledge translation methods in cardiac arrest,°’ a nutritional 
intervention to reduce weight loss in hospital patients,°° and the treatment of sexually 
transmitted disease.°° The primary benefit of a stepped wedge design comes in the 
implementation phase. Since all study sites will have been introduced to the intervention 
during the testing phase, the first step in implementing the new practice and training 
practitioners in using the new practice have already been completed at all of the sites. 
More information and a presentation on designing stepped wedge studies and how to 
analyze the data are available on AHRQ’s PBRN Resource Center Web site at 
http://pbrn.ahrg.gov/events/advanced-methods-primary-care-research-stepped-wedge- 


design.©° 
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5. Adaptive Trial Designs 


Adaptive trial designs are RCTs that have a process of adaptation of the intervention 
designed into the study. These trials can be briefer and require a smaller sample size 
depending on the specific adaptations that are proposed. Adaptive trial designs plan for 
adaptation either at certain endpoints or based on certain findings.*’ Many adaptive trial 
designs use Bayesian analysis to adjust the randomization process based on prior 
findings (as was used in the Fiore study*® above). There are at least 10 different types of 
adaptive trial designs,°* and PCORI offers a set of standards for conducting adaptive 
Clinical trials (PDF — 381 KB). 


3.5.4 Accelerating RCTs 

PBRNs and other research networks are well positioned to accelerate the process of 
conducting an RCT. One of the ways that researchers inadvertently extend the time that a 
clinical trial takes to recruit subjects is by asking a question that is confined to too narrow a 
population. Researchers must balance the ability to have statistically significant differences 
between the treatment and control group by limiting the population under study and 
developing treatments and protocols that can be generalized to real treatment populations. 
Following the steps in this document’s preparation, problem identification and knowledge 
exploration sections will help researchers design studies that can recruit adequate samples in 
reasonable time, particularly when multiple sites are used for recruitment. 


Following the processes described above for engaging practices and patients in all aspects 
of study design should also help research teams in other ways. They avoid developing study 
protocols that are difficult to implement or require significant investments in training and audit 
of whether the practice is being implemented with fidelity. By working to fully understand the 
patient’s and the primary care provider's needs, researchers can design interventions that 
can be adopted within existing workflows and address problems that practices would like to 
solve, thereby increasing the probability of dissemination. 


3.6 Implementation and Dissemination Phases 


3.6.1 Implementation 


Once a research or a QI project has demonstrated results, ensuring implementation within 
the organization(s) that conducted the testing can be the first step in a dissemination 
process. The reasons that the practices involved in the research sometimes do not adopt 
recommendations based on the outcomes of studies in which they were involved are often 
related to concerns that can be addressed using the tools in this document's preparation, 
problem identification, solution-finding, and testing sections. Questions about whether the 
patients in the practice are generally like the patients in the study or whether the study 
patients were a small sub group (e.g., they had heart disease but no other chronic 
conditions), whether the clinicians involved were really representative of the population of 
physicians expected to implement the practice, and how the clinical and administrative 
leadership feel about the change can all affect adoption of a new practice, even within 
organizations that successfully tested it. These factors may loom even larger in 
organizations that are introduced to a new practice without having been involved in the study. 
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Implementation is another area in which the RE-AIM framework may prove useful. Clinical 
studies often do not adequately address organizational barriers to adoption of a new practice 
and therefore underestimate the ease with which a new practice will be adopted. By 
considering organizational issues during the early phases of planning a study and selecting a 
solution, the Ql/research team will be better prepared for implementation when they have a 
positive outcome to their study. Study designs that involve implementation activities as part 
of the testing protocol, such as the Bayesian randomization algorithm used in the Fiore”? 
study in the VA (described above), can move an organization toward adoption during the 
study process. 


Problems also arise because of assumptions that good practices will automatically be 
adopted. In fact there is overwhelming evidence that most people and organizations are 
highly risk averse and will not change practice even in the face of overwhelming evidence 
that the change is an improvement.®? Consequently, clinical settings do not prepare to adopt 
new practices. Informing/educating appropriate staff is only one step. Changes in 
organizational processes may also be necessary. A tool to assess an organization’s 
readiness to implement an innovation may identify the organizational facilitators and barriers 
that need to be addressed to successfully implement an innovation. One tool that was 
created for health care providers™ is available at http://networkofpractice.org/ris/getstarted/. 
One of the benefits of a PBRN or of a large integrated health system is the availability of 
practice facilitators who use QI techniques and implementation science to ensure fidelity in 
the adoption of new practices. For additional information and resources, download AHRQ’s 
Developing and Running a Primary Care Practice Facilitation Program: A How-to Guide 
accessible here (PDF — 2.9 MB) 


Sustaining a practice improvement is not guaranteed even once it is implemented. One tool 
that is available to assess an organization’s ability to sustain a practice change once it has 
been adopted is the sustainability index® developed for the British National Health Service. 
The tool is available in an electronic format at 

http://www. niatx.net/Content/ContentPage.aspx?NID=156. This model outlines three levels 
of intervention and support that are necessary for a new practice to be sustained in an 
organization: (1) staff including leadership, (2) organizational structure and culture, and (3) 
the process including the evidence base of the new intervention and the process of 
implementing it. It uses a simple 10-question survey to assess strengths and weaknesses of 
a specific change within a health care practice. 


External factors may affect whether a new practice is adopted. Regulation, funding, and 
perception of peers may all affect whether and to what extent an organization adopts a 
particular practice. While academic researchers may not think that addressing these levers 
of change is under their purview, other industries consider these things critical to ensuring 
that their products and services are adopted. 


3.6.2 Dissemination 


Researchers are expected to disseminate the findings of their projects. While publication in 
peer-reviewed journals is often the goal of academic scientists, publication in scientific 
journals rarely promotes the dissemination of findings beyond the small circle of researchers 
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interested in the topic. Funders now expect researchers to have a plan for how study 
findings will be disseminated to people who can use them. 


There are many ways to disseminate information in today’s electronically connected world. 
The three most common types of dissemination used by PBRNs are (1) publications in peer- 
reviewed medical journals; (2) professional society meetings; and (3) local, regional, national, 
and international meetings. Word of mouth using stories is a powerful means of 
dissemination that is frequently disregarded. Sometimes natural diffusion via word of mouth 
just happens. More often than not, an active process using tools such as influence analysis 
and social network mapping to facilitate word of mouth dissemination is necessary. Stories 
from a patient or provider perspective are often a more effective means of communicating 
information than scientific presentations of study outcomes. For most audiences, data is 
better used as a way to back up a claim demonstrated by an anecdote. University extension 
offices and marketers of commercial products and services have developed a deep 
understanding of how to use influential social network members to accelerate dissemination 
of innovations via word of mouth. Research teams may benefit from contacting their local 
public university extension office for guidance. 


While academic research teams and organizational Ql/research staff generally lack the 
resources to actively disseminate their findings on a large scale, there are mechanisms to 
help health care providers adopt new practices. For example, the National Area Health 
Education Centers (AHEC) Organization has resources on their Web site here: 
http://www.nationalahec.org/ to help disseminate and implement new practice guidelines in 
health care practices. Researchers who wish to disseminate new discoveries to practices 
beyond their research network may find that the technology transfer centers funded by HHS, 
such as the AHECs, have tools and resources to support their efforts. 


On Wednesday, July 22nd 2015, the AHRQ 


PBRN Resource Center hosted Dissemination DISSEMINATION AND 
and Implementation Research in Health: IMPLEMENTATION 
Translating Science to Practice authors Drs. RESEARCH IN HEALTH 


Ross Brownson and Graham Colditz ona Transtating Science to Practice 
Webinar to discuss their new book. View the 
Webinar recording on AHRQ’s PBRN Events 
page here: 
https://obrn.ahrg.gov/events/dissemination- 
and-implementation-research-health- 
translating-science-practice™. 
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Innovation and Improvement. 2007. 

http:/Awww.institute.nhs.uk/index.php?option=com joomcart&ltemid=194&main page=d 





ocument product info&cPath=67&products id=290&Joomcartid=2ofvOdookq/c4dfpirss 
3dmvi5. 
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66. Brownson RC, Colditz GA, Proctor EK. Dissemination and Implementation Research in 
Health: Translating Science to Practice. AHRQ PBRN Resource Center Webinar. 2015. 


https://obrn.ahrg.gov/events/dissemination-and-implementation-research-health- 
translating-science-practice. 
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Appendix B. Annotated Bibliography of Select Publications on “Rapid- 
Cycle” Research 


This appendix presents an annotated bibliography of publications relevant to rapid-cycle 
research in the PBRN context. The term “rapid-cycle research” may mean different things to 
different people, and other terminology is often used to describe the concept of conducting 
studies that can be completed quickly and produce results that are easily incorporated into 
clinical practice. This annotated bibliography lists select articles on rapid-cycle research, 


quality improvement, and the different phases of Dr. Gustafson’s “Secrets of Rapid 


Research” Framework (see Exhibit 1.1.) 


PN alaleye-\(ctomm J] e) (cele le-] lah me) mel-)(2(e1 a (ctVolelger-t-m ohVmat-|e)(ebeael(-m-lare| 
Quality Improvement Phase 


Citation 


Summary 





Resources Relevant to Quality Improvement 


Gustafson D, Johnson K The NIATx Model: 
Process Improvement in Behavioral Health. 
Madison: University of Wisconsin; 2011. 
Funded by: The authors received no financial 
support for the research, authorship, and/or 
publication of this book. 


This book outlines the simple process improvement 
model developed at NIATx by Dr. Gustafson and 
his colleagues. 





Plsek PE. Quality improvement methods in 
clinical medicine. Pediatrics 
1999;103(Supplement E1):203-14. PMID: 
9917464. 

Funded by: The author received no financial 
support for the research, authorship, and/or 
publication of this article. 


This article surveys the methods and tools of 
quality improvement used today in health care. 
Specifically, the article describes how clinicians can 
use these methods to impact the clinical practice of 
medicine. 





Starr SR, Kautz JM, Sorita A, et al. Quality 
improvement education for health 
professionals: a systematic review. Am J Med 
Qual 2015 Jan. PMID: 25583877. 
http://www.ncbi.nim.nih.gov/pubmed/25583877. 
Funded by: The authors received no financial 
support for the research, authorship, and/or 
publication of this article. 





This systematic review describes the use of clinical 
measures in quality improvement education for 
health professionals. The research team used 
clinical measures to evaluate the prevalence of Ql 
curricula, and their association with several 
curricular features. Though effective QI education 
should measurably improve patient care, the 
research team found that little more than half of the 
published QI curricula studies included clinical 
measures. Health care curricula that predicted 
success included learner buy-in, adequate teacher 
expertise, role-modeling, mixed teaching methods, 
adequate curricular time to finish a project, anda 
supportive institutional culture. 

[Also relevant for: implementation phase] 
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ferirliceya 


Summary 


Resources Relevant to the Preparation Phase 


Baldwin L-M, Graham D, Schmit K, et al. 
Toolkit for Developing and Conducting Multi- 
site Clinical Trials in Practice Based Research 
Networks. 
http://www.dartnet.info/clinicaltrialsPBRNtoolkit. 
htm. 

Funded by: This toolkit was developed in 
collaboration with the DARTNet Institute 
Research Steering Committee and the Clinical 
Translational Science Award (CTSA) Practice 
Based Research Network Workgroup of the 
Community Engagement Key Function 
Committee. 


This toolkit offers considerations for researchers 
interested in conducting clinical trials ina PBRN 
setting. Questions related to clinical effectiveness, 
comparative effectiveness, and safety are 
addressed. 





Gold M, Taylor EF. Moving research into 
practice: lessons from the US Agency for 
Healthcare Research and Quality's IDSRN 
program. Implement Sci 2007 Mar;29:2-9. 
PMID: 17394644. 

Funded by: The research upon which this 
article is based was supported by a contract 
from the Agency for Healthcare Research and 
Quality of the U.S. Department of Health and 
Human Services. 


This article describes an evaluation of AHRQ's 
Integrated Delivery Systems Research Network 
(IDSRN) program. Interviews and case studies 
helped identify several factors import to and limiting 
success of projects in influencing operations. 





Thorpe K. PRECIS Tool: Understanding your 
Research Intentions, the Pragmatic- 
Explanatory Continuum. AHRQ PBRN 
Resource Center Webinar. 2014. 
http://pbrn.ahrq.gov/events/precis-tool- 
understanding-your-research-intentions- 


pragmatic-explanatory-continuum. 
Funded by: AHRQ Contract No. HHSA290- 


2010-00004I, Task Order No. 32006T 
Resources Relevant to th 

Bergman D, Arnetz B, Wahlstr6m R, et al. 

Effects of dialogue groups on physicians' work 

environment. J Health Organ Manag 

2007;21(1):27-38. PMID: 17455810. 

Funded by: Funding source unknown. 


This Webinar details the pragmatic-explanatory 
continuum indicator summary (PRECIS) tool. It 
focuses on the development of the tool and the 
methods for its use by trialists to determine whether 
one’s research design decisions are consistent with 
the stated purpose of one’s research trial. 





e Problem Exploration Phase 

This article examines whether dialogue groups for 
physicians can improve their psychosocial work 
environment. 





Bhasale AL, Miller GC, Reid SE et al. 
Analysing potential harm in Australian general 
practice: an incident-monitoring study. Med J 
Aust, 1998 Jul;169(2):73-6. PMID: 9700340. 
Funded by: The Commonwealth Review of 
Professional Indemnity Arrangements for 
Health Care Professionals. 


This article is an observational study of incidents of 
potential harm based on a modified critical incident 
technique. A non-random sample of general 
practitioners anonymously submitted incident 
reports contemporaneously. 





Bohm D. On dialogue. New York: Routledge; 
1996. http://www.amazon.com/Dialoque- 
Routledge-Classics-David- 


Bohm/dp/0415336414. 
Funded by: The authors reported no funding 


assistance for the preparation of this book. 


This is the original text describing the dialogue 
technique developed by Bohm. 
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OTE LiCoy a) 
Delbecg AL, Van deVen AH. A Group Process 
Model for Problem Identification and Program 
Planning. J App! Behav Sci 1971;July/August 
7(4):466-92. 
Funded by: The authors reported no funding 
assistance for the preparation of this article. 


Summary 
This article demonstrates the application of group 
process approach by administrators for program 
planning. 








Fitzsimmons JA, Maurer GB. A walk-through 
audit to improve restaurant performance. The 
Cornell Hotel and Restaurant Administration 
Quarterly 1991 Feb;31(4):95-100. 
http://cqx.sagepub.com/content/31/4/95. 
Funded by: The authors express their 
appreciation to Kent Hughes and the 

Texas Restaurant Association for supporting 
this project, and to Mona J. Fitzsimmons for 
her assistance. 


This article reports on the findings of a "walk- 
through audit" for a restaurant. The audit, which 
consists of 42 questions, was developed as a 
management tool for the systematic evaluation of a 
customer's view of the service provided in a 
restaurant. 





Flanagan JC. The critical incident technique. 
Psychol Bull 1954 Jul;51(4):327-58. PMID: 
13177800. 

Funded by: The authors reported no funding 
assistance for the preparation of this 
manuscript. 


This article discusses the development, 
fundamental principles, and original uses of the 
critical incident technique, along with a review of 
studies employing the technique and suggestions 
for further applications. 





Ford JH, Green CA, Hoffman KA et al. Process 
improvement needs in substance abuse 
treatment: Admissions walkthrough results. J 
Subst Abuse Treat 2007;33(4):379-89. PMID: 
17499961. 

Funded by: The Robert Wood Johnson 
Foundation and the Center for Substance 
Abuse Treatment support the Network for the 
Improvement of Addiction Treatment through 
grants and cooperative agreements. Awards to 
the University of Wisconsin (RWJF 48364; 
CSAT-SC-04-035) and Oregon Health & 
Science University (RWJF 46876 and 50165; 
PIC-STAR-SC-03-044) supported the 
preparation of this manuscript. 


This article outlines the findings from walk-throughs 
of admission processes for 327 addiction treatment 
agencies in the U.S. 





Gustafson DH, McTavish F, Hawkins R, et al. 
Computer support for elderly women with 
breast cancer. JAMA 1998 
October;280(15):1305. PMID: 9794300. 
Funded by: Funding source unknown. 





This letter to the editor of JAMA magazine 
describes an evaluation of patient education 
through use of a personal computer at home by 
elderly cancer patients. 
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Citation 
How to Use the Nominal Group Technique. 
NIATx. 
http://www. niatx.net/content/contentpage.aspx? 
NID=147 
Funded by: The University of Wisconsin- 
Madison Center for Health Enhancement 
Systems Studies (CHESS). 


Summary 
This Web site states and describes the 7 steps of 
the nominal group technique. 








Isaacs W. Dialogue: The art of thinking 
together. Crown Business; 2008. 
http://www.randomhouse.com/book/85559/dial 
ogue-by-william-isaacs/9780307483782/. 
Funded by: The authors reported no funding 
assistance for the preparation of this book. 


This book further describes the dialogue technique 
first described by Bohm. 





Ilvarsson B, Larsson S, Sjéberg T. Patients' 
experiences of support while waiting for cardiac 
surgery. a critical incident technique analysis. 
Eur J Cardiovasc Nurs 2004;3(2):183-91. 
Funded by: Funding source unknown. 


This article discusses how identifying factors that 
influence patients' experiences of support while 
they wait for heart surgery using a CIT, institutional 
and non-institutional health care services can 
improve the organization of the entire health care 
process and develop patient-focused support 
programs. 





Kvarnstrém S. Difficulties in collaboration: A 
critical incident study of interprofessional 
healthcare teamwork. J Interprof Care 
2008;22(2):191-203. 

Funded by: Funding source unknown. 


This article discusses 18 individual interviews using 
a Critical Incident Technique. These were 
performed with 18 Swedish professionals working 
in health care teams and were examined with 
qualitative content analysis. The main findings 
show difficulties related to the team dynamic that 
arose when team members acted towards one 
another as representatives of their professions and 
difficulties that occurred when the members' 
various knowledge contributions interacted in the 
team. 





Leonard-Barton D. The factory as a learning 
laboratory. Sloan Management Review 1992 
Oct;34:23. 
http://sloanreview.mit.edu/article/the-factory-as- 


a-learning-laboratory/. 
Funded by: The study was supported by the 


Division of Research, Harvard Business 
School. 


This article discusses distinguishing activities 
critical to a learning laboratory: (1) problem solving, 
(2) internal knowledge integration, (3) innovation 
and experimentation, and (4) integration of external 
information flows. 





Maemura Y. Changing stereotypes in India's 
garment sector through dialogue. J Conflict 
Resolut 201 3;31(2):159-187. 

Funded by: The authors reported no funding 
assistance for the preparation of this 
manuscript. 





This article reports on a methodology for productive 
dialogue and collaboration within groups 
experiencing tension. 
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Citation 
McCarty D, Gustafson DH, Wisdom JP, et al. 
The Network for the Improvement of Addiction 
Treatment (NIATx): enhancing access and 
retention. Drug Alcohol Depend 
2007;88(2):138-45. PMID: 17129680. 
Funded by: The Network for the Improvement 
of Addiction Treatment (NIATx) was supported 
through grants from the Robert Wood Johnson 
Foundation and cooperative agreements from 
the Substance Abuse and Mental Health 
Services Administration, Center for Substance 
Abuse Treatment. The National Evaluation 
Team at Oregon Health and Science University 
was supported through awards from the Robert 
Wood Johnson Foundation (46876 and 50165), 
the Center for Substance Abuse Treatment 
(through subcontracts from Northrop Grumman 
Corporation -- PIC-STAR-SC-03-044, 
SAMHSA SC-05-110), and the National 
Institute on Drug Abuse (RO1 DA018282). 
National Program Office activities at the 
University of Wisconsin were supported 
through awards from the Robert Wood 
Johnson Foundation (48364), and the Center 
for Substance Abuse Treatment (through a 
subcontract from Northrop Grumman 
Corporation -- PIC-STAR-SC-04-035). 


Summary 
This article discusses how small incremental 
changes in treatment processes can lead to 
significant reductions in days to treatment and 
consistent gains in retention in addiction treatment. 





Nagda BA, Spearmon ML, Holley LC, et al. 
Intergroup dialogues: An innovative approach 
to teaching about diversity and justice in social 
work programs. J Soc Work Educ 
1999;35(3):433-449. 

Funded by: The project reported in this article 
was funded by a Council on Social Work 
Education Millennium Project grant and the 
University of Washington School of Social 
Work. 


This article reports on intergroup dialogues and 
suggests ways to incorporate them into social work 
program curricula. 





Ridings A. Pause for Breath: Bringing the 
Practices of Mindfulness and Dialogue to 
Leadership Conversations. London: Live It 
Publishing; 2011. 
http://www.amazon.com/Pause-Breath- 
Mindfulness-Leadership- 
Conversations/dp/1906954232. 

Funded by: The authors reported no funding 
assistance for the preparation of this book. 





This book further describes the dialogue process 
and adds the element of mindfulness to the 
practice. 
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Citation 
Van de Ven AH, Delbecg AL. The nominal 
group as a research instrument for exploratory 
health studies. Am J Public Health 
1972;62(3):337-42. PMID: 5011164. 
Funded by: The authors reported no funding 
assistance for the preparation of this 
manuscript. 


Summary 
This paper sets forth a group process for qualitative 
judgmental problem exploration that is particularly 
applicable to the subjective and judgmental 
character of many health planning efforts. 


Resources Relevant to the Solution Development Phase 


Langley GJ, Moen R, Nolan KM, et al. The 
improvement guide: a practical approach to 
enhancing organizational performance. 2nd ed. 
San Francisco: Jossey-Bass; 2009. 

Funded by: The authors reported no funding 
assistance for the preparation of this book. 


This text provides tools, techniquesb and 
explanations for how to conduct process 
improvement activities to improve organizational 
performance. 

[Also relevant for: all phases, as a general 
resource] 





Rogers EM. Diffusion of Innovations. 5th ed. 
New York: Simon and Schuster; 2003. 
http://www.amazon.com/Diffusion-Innovations- 


Edition-Everett-Rogers/dp/0743222091. 
Funded by: The authors reported no funding 


assistance for the preparation of this book. 


In this text, Rogers explores the research on 
innovation diffusion and dissemination. 





Senge P. The fifth discipline: The art & practice 
of the learning organization. United States: 
Doubleday; 1990. 
http://www.amazon.com/The-Fifth-Discipline- 


Practice-Organization/dp/038551 7254. 
Funded by: The author reported no funding 


assistance for the preparation of this book. 

Resources Relevant to 
Benneyan J, Lloyd R, Plsek P. Statistical 
process control as a tool for research and 
healthcare improvement. Qual Saf Health Care 
2003 Dec;12(6):458-64. PMID: 14645763. 
Funded by: Funding source unknown. 


This text outlines 11 laws of learning organization, 
“the fifth discipline’. 





the Solution Testing Phase 

This article describes statistical process control and 
its primary tool, the control chart, along with several 
practical health care applications, in the context of 
improvement of process performance 
measurement. 





Berry DA. Adaptive clinical trials: the promise 
and the caution. J Clin Oncol 201 1;29(6):606-9. 
PMID: 21172875. 

Funded by: The author reported no funding 
assistance for the preparation of this 
manuscript 


This article analyses the advantages and 
disadvantages of using adaptive randomization 
with a Bayesian perspective in multi-arm clinical 
trials. 





Biglan A, Ary D, Wagenaar AC. The value of 
interrupted time-series experiments for 
community intervention research. Prev Sci 
2000;1(1):31-49. PMID:11507793. 

Funded by: This paper was supported by 
Grants DA09306 and DA 09678 from the 
National Institute of Drug Abuse and Grant No. 
CA38273 from the National Cancer Institute. 


This paper reports on the uses of multiple baseline 
time-series designs and related repeated-measures 
time series experiments as they relate to the 
evaluation of community interventions and policy 
changes. Threats to internal validity are also 
described. 
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Citation 
Birtwhistle R. How can PBRNs use Big Data to 
Create a Learning Health System: The 
Canadian Primary Care Sentinel Surveillance 
Network (CPCSSN). Proceedings of North 
American Primary Care Research Group 
(NAPCRG) Practice-based Research Network 
Conference; 2014 Jun 30- Jul 1; Bethesda. 
Bethesda: North American Primary Care 
Research Group; 2014. (PDF — 1.38 MB). 
Funded by: CPCSSN is funded by the Public 
Health Agency of Canada, Health Canada and 
Canada Health Infoway and the Canadian 
Institute for Health Research. 


Summary 
This presentation describes how PBRNs can use 
‘big data’ in creating a learning health system and 
discusses three uses of CPCSSN EMR data. 





Campbell DT, Stanley JC, Gage, NL. 
Experimental and Quasi-Experimental Designs 
for Research. Boston: Houghton Mifflin 
Company; 1963. (PDF — 6.64 MB). 

Funded by: The authors reported no funding 
assistance for the preparation of this book. 


This is the classic text on experimental and quasi- 
experimental design; it outlines the various ways of 
setting up a study, what assumptions are made 
with each design, and what conclusions can be 
drawn from each design. 





Chakraborty B, Collins LM, Strecher VJ, et al. 
Developing multicomponent interventions using 
fractional factorial designs. Stat Med 
2009;28(21):2687-708. PMID:19575485. 
Funded by: Support for this project was 
received from the National Institutes of Health 
grants RO1 MH080015, P50 DA10075, and 
P50 CA101451. 


This report operationalizes screening studies and 
evaluates the efficacy of using factorial designs to 
screen out the least active component of 
multicomponent interventions. 





Chow SC, Chang M. Adaptive design methods 
in clinical trials - a review. Orphanet Journal of 
Rare Diseases 2008 May;2:3-11. PMID: 
18454853. 

Funded by: The authors reported no funding 
assistance for the preparation of this journal 
article. 


This article reviews the adaptive trial design and 
briefly describes all 10 ways in which trials have 
been adapted 





Dainty KN, Scales DC, Brooks SC, et al. A 
knowledge translation collaborative to improve 
the use of therapeutic hypothermia in post- 
cardiac arrest patients: protocol for a stepped 
wedge randomized trial. Implement Sci. 
2011;6:4. PMID: 21235799. 

Funded by: This study was funded by the 
Heart and Stroke Foundation of Canada, the 
Canadian Institutes of Health Research, and by 
the Laerdal Foundation for Acute Medicine 
(Norway). The trial is registered under 
ClinicalTrials.gov Trial Identifier: 
NCT00683683. 





This article outlines a proposed stepped wedge 
randomized trial design to evaluate a knowledge 
translation strategy designed to increase the 
utilization rate of induced hypothermia in survivors 
of cardiac arrest across a network of Canadian 
hospitals. 
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Citation 
D'Avolio L, Ferguson R, Goryachev S, et al. 
Implementation of the Department of Veterans 
Affairs’ first point-of-care clinical trial. J Am Med 
Inform Assoc 2012;19(e1), e170-176. PMID: 
22366293. 
Funded by: This work was supported by the 
VA Office of Research and Development and 
the VA Cooperative Studies Program. Phillip 
Lavori was supported by grants UL1 
RR025744 and RO1 MH051481 to Stanford 
University. 


Summary 
This article describes the development of the point- 
of-care clinical trial methods. 








Design and Implementation of N-of-One Trials: 
A User's Guide. Final. Prepared by Brigham 
and Women’s Hospital DEcIDE Methods 
Center under Contract No. 290-2005-0016-I. 
AHRQ Publication No. 13(14)-EHC122-EF. 
Rockville, MD: Agency for Healthcare 
Research and Quality; February 2014. (PDF — 
900 KB). 

Funded by: AHRQ Contract No. 290-2005- 
0016-1. 





This AHRQ-developed tool outlines the purposes of 
n-of-one trials, the best uses of this methodology 
and provides some examples and tools for deciding 
whether and how to conduct n-of-one studies 





Detry MA, Lewis RJ, Broglio KR, et al. 
Standards for the Design, Conduct, and 
Evaluation of 

Adaptive Randomized Clinical Trials. 
http://www.pcori.org/assets/Standards-for-the- 
Design-Conduct-and-Evaluation-of-Adaptive- 


Randomized-Clinical-Trials.pdf. 
Funded by: The Patient-Centered Outcomes 


Research Institute (PCORI) through a contract 
to support the Methodology Committee’s 
development of a report to outline existing 
methodologies for conducting patient centered 
outcomes research, propose appropriate 
methodological standards, and identify 
important methodological gaps that need to be 
addressed. 


This document contains standards for conducting 
adaptive trials in terms of study design, analysis, 
and implementation. 





Diaz M, Neuhauser D. Pasteur and parachutes: 
when statistical process control is better than a 
randomized controlled trial. Qual Saf Health 
Care 2005 Apr;14(2):140-3. 

Funded by: This article was funded by the 
Charles Elton Blanchard MD Endowment, 
Medical School, Case Western Reserve 
University, Cleveland, OH. 





This article demonstrates the application of 
statistical process control by use of control charts in 
several examples, including one regarding rabies 
vaccination and another about parachute jumping. 
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Citation 
Dickinson M, Bartlett-Esquilant G, Kwan B, et 
al. Advanced Methods for Primary Care 
Research: The Stepped Wedge Design. Video. 
AHRQ PBRN Resource Center Webinar. 2015. 
http://pbrn.ahrg.gov/events/advanced-methods- 


primary-care-research-stepped-wedge-design. 
Funded by: AHRQ Contract No. HHSA290- 


2010-000041, Task Order No. 32006T 


Summary 
This Web site provides a number of articles 
describing the pros and cons of stepped wedge 
design, when to use a stepped wedge design and 
studies that are using this study design. It also has 
slides and a link to a presentation on how to do 
stepped wedge studies, including setting up the 
analysis. 








Donovan EF, Lannon C, Bailit J, et al. A 
statewide initiative to reduce inappropriate 
scheduled births at 36 0/7—38 6/7 weeks' 
gestation. Am J Obstet Gynecol 
2010;202(3):243.e1-8. PMID: 20207241. 
Funded by: This article was supported in part 
by Grant No. 1U0CMS030227/01 from the 
Center for Medicare & Medicaid Services 
(CMS) administered by the Ohio Department of 
Job and Family Services. 


This article describes the effect of a statewide 
quality collaborative, specifically Institute for 
Healthcare Improvement Breakthrough Series 
interventions, in reducing the number of scheduled 
births lacking a documented medical indication. 





Eslami 1, Abu-Hanna A, de Keizer NF, et al. 
Implementing glucose control in intensive care: 
a multicenter trial using statistical process 
control. Intensive Care Med 2010 
Sep;36(9):1556-65. PMID: 20533024. 

Funded by: Funding source unknown. 


This article describes a study that investigated 
glucose control (GC) performance over time in 
three intensive care units and in routine clinical 
practice during implementation of GC strategies. 





Fiore LD, Brophy M, Ferguson R, et al. A point- 
of-care clinical trial comparing insulin 
administered using a sliding scale versus a 
weight-based regimen. Clin Trials 

201 1;8(2):183-95. PMID: 21478329 

Funded by: This study was supported by the 
Department of Veterans Affairs Cooperative 
Studies Program. Dr Lavori’s efforts have been 
supported by grants UL1 RR025744 and RO1 
MH051481 to Stanford University. 


This article describes a point-of-care randomized 
controlled trial to improve diabetes treatment. 





Glasgow RE, Klesges LM, Dzewaltowski DM, 
et al. The future of health behavior change 
research: what is needed to improve translation 
of research into health promotion practice? Ann 
Behav Med 2004;27(1):3-12. PMID: 14979858. 
Funded by: National Cancer Institute, Agency 
for Healthcare Research and Quality, and NIH 
Office of Behavioral and Social Science 
Research. 





This article describes how to use the RE-AIM 
model for assessing studies and has a set of 
questions for researchers and research funders to 
consider to improve the use of research findings in 
practice 
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Citation 
Glasgow RE., Lichtenstein E, Marcus AC. Why 
don't we see more translation of health 
promotion research to practice? Rethinking the 
efficacy-to-effectiveness transition. Am J Public 
Health 2003 August;93(8):1261—-67. PMID: 
12893608. 
Funded by: This project was funded by The 
Robert Wood Johnson Foundation (grant 
030102) and the Agency for Healthcare 
Research and Quality (grant HS10123). 


Summary 
This article offers recommendations to help close 
the gap between research and practice by 
addressing one of the reasons for this gap: the 
assumption that successful efficacy research 
naturally leads to effectiveness research. The 
article recommends greater attention be paid to 
documentation to guide evaluation and possible 
adoption of new programs. 








Golden MR, Kerani RP, Stenger M, et al. 
Uptake and population-level impact of 
expedited partner therapy (EPT) on chlamydia 
trachomatis and neisseria gonorrhoeae: The 
Washington State Community-Level 
Randomized Trial of EPT. PLoS Med 
2015;12(1):e1001777. PMID: 25590331. 
Funded by: The authors reported no funding 
assistance for the preparation of this 
manuscript. 


This article discusses the methods and findings of 
a stepped-wedge, community-level randomized trial 
to evaluate the efficacy of a public health 
intervention promoting expedited partner therapy to 
decrease chlamydia test positivity and gonorrhea 
incidence in women. 





Hanrahan L. Clinical Big Data and Practice- 
Based Research Networks are the Foundations 
for a Rapid Learning Health System. 
Proceedings of North American Primary Care 
Research Group (NAPCRG) Practice-based 
Research Network Conference; 2014 Jun 30— 
Jul 1; Bethesda. Bethesda: North American 
Primary Care Research Group; 2014. (PDF — 
1.88 MB). 

Funded by: This project was supported 
through the University of Wisconsin School of 
Medicine and Public Health Block Grant Fund. 


This presentation discusses the use of health 
systems intelligence and analytics to provide 
advice for how PBRNs can use big data for 
creation of a learning health system. 





Harrington G, Watson K, Bailey M, et al. 
Reduction in hospitalwide incidence of infection 
or colonization with methicillin-resistant 
Staphylococcus aureus with use of 
antimicrobial hand-hygiene gel and statistical 
process control charts. Infect Control Hosp 
Epidemiol 2007 Jul;28(7):837-44. PMID: 
17564987. 

Funded by: This article was funded in part by 
Les Entreprises Solumed, Medical Specialities 
Australia, and Ansell Limited. 


This article describes a longitudinal observational 
study in which a series of interventions were 
carried out to reduce the incidence of methicillin- 
resistant Staphylococcus aureus. 





Harvey G, Kitson A. Implementing Evidence- 
Based Practice in Healthcare: A Facilitation 
Guide. New York: Routledge; 2015. 

Funded by: The authors reported no funding 
assistance for the preparation of this book. 





This book delves into possible facilitation methods 
to use across health care settings, including real- 
time implementation projects. 
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OTE LiCey a) 
Improving Care Delivery Through Lean: 
Implementation Case Studies: Contract Report. 
November 2014. Agency for Healthcare 
Research and Quality, Rockville, MD. 
http://www. ahrg.gov/professionals/systems/syst 


em/systemdesign/leancasestudies/index.htm 
Funded by: AHRQ Contract No. 


HHSA290200600019 


Summary 
This AHRQ-funded report includes six in-depth 
case studies that explain how Lean principles were 
applied in 13 implementation projects. These 
included: improving patient flow during hospital 
care, electronic prescribing of medicines, reducing 
the cost of hip and knee replacement surgery, and 
preventing urinary tract infection, amongst others. 
[Also relevant for: implementation phase] 





Institute for Healthcare Improvement. How to 
Improve: IHI Model for Improvement. 
http://www. ihi.org/resources/Pages/Howtolmpr 


ove/default.aspx. 
Funded by: This Web site is funded by the 


Institute for Healthcare Improvement. 


This Web site outlines the Model for Improvement, 
developed by Associates in Process Improvement 
to accelerate improvement. The plan-do-study-act 
cycles are outlined as an example of how to test 
changes on a small scale. 

[Also relevant for: implementation and 
dissemination phase] 





Lauer MS, D'Agostino RB, Sr. The randomized 
registry trial--the next disruptive technology in 
Clinical research? N Engl J Med 
2013;369(17):1579-81. 

Funded by: The authors reported no funding 
assistance for the preparation of this journal 
article. 


This article discusses the possibilities for using 
EHR data to recruit participants and assess study 
outcomes in an RCT with fewer resources, and ina 
shorter time than is typical but adds a caution about 
data quality. 





Lloyd R. On Demand: Using Run and Control 
Charts to Understand Variation. Video. 
http://www. ihi.org/education/WebTraining/OnD 


emand/Run_ControlCharts/Pages/default.aspx. 
Funded by: This Web site is funded by the 


Institute for Healthcare Improvement. 


This link is to the Institute for Healthcare 
Improvement's online course in using statistical 
process control tools for process improvement and 
research 





Pace W. Linking PBRN Research and Big 
Data: The DARTNet Experience. Proceedings 
of North American Primary Care Research 
Group (NAPCRG) Practice-based Research 
Network Conference; 2014 Jun 30- Jul 1; 
Bethesda. Bethesda: North American Primary 
Care Research Group; 2014. (PDF — 900 KB). 
Funded by: The authors reported no funding 
assistance for the preparation of this 
presentation. 


This presentation utilizes DARTNet as a case study 
to describe how big data can be linked to patient 
outcomes and provider decision making in PBRN 
research. 





Shewhart WA. Economic control of quality of 
manufactured product (Vol. 509). ASQ Quality 
Press; 1931. 

Funded by: The authors reported no funding 
assistance for the preparation of this book. 


This book discusses the control chart and the 
principles behind it, which draw from the disciplines 
of statistics, engineering, and economics. 





Shrank W. The Center For Medicare and 
Medicaid Innovation’s blueprint for rapid-cycle 
evaluation of new care and payment models. 
Health Aff (Millwood) 2013 ;32(4):807-12. 
Funded by: The authors reported no funding 
assistance for the preparation of this journal 
article. 





This article describes the approaches this group 
will take as it evaluates new delivery and payment 
models launched by the Innovation Center under 
the Affordable Care Act. 
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Citation 
Smith JD. Single-case experimental designs: A 
systematic review of published research and 
current standards. Psychol Methods 
2012;17(4):510-50. PMID: 22845874. 
Funded by: Research support for the author 
was provided by research training grant 
MH20012 from the National Institute of Mental 
Health, awarded to Elizabeth A. Stormshak. 


Summary 
This article describes the research design, 
measurement, and analysis domains distinctive to 
the single-case experimental design (SCED) and 
provides updated benchmarks for key SCED 
characteristics. 





Strecher VJ, McClure JB, Alexander GL, et al. 
Web-based smoking-cessation programs: 
results of a randomized trial. Am J Prev Med 
2008;34(5):373-81. PMID: 18407003 

Funded by: This grant was funded from 
National Cancer Institute grants P50 
CA101451 and RO1 CA101843. This project 
was conducted in collaboration with the Cancer 
Research Network (CRN). The CRN is a 
consortium of research organizations affiliated 
with nonprofit integrated health care delivery 
systems and the National Cancer Institute. 
Nicotine replacement therapy was provided by 
GlaxoSmithKline. 


This article discusses the use of a novel fractional 
factorial design to evaluate study aims relating to 
improving smoking cessation interventions. 
Fractional factorial designs will be important for 
evaluating eHealth activities moving forward. 





The Pennsylvania State University. Introduction 
to Factorial Experimental Designs. 
http://methodology.psu.edu/ra/most/factorial. 
Funded by: This Web site is funded by the 
Institute for Healthcare Improvement. 

Virginia Polytechnic Institute and State 
University. Reach Effectiveness Adoption 
Implementation Maintenance (RE-AIM). 
http://www.re-aim.hnfe.vt.edu/index.html. 
Funded by: This Web site is funded by Virginia 
Tech. 


This Web site provides a primer for conducting 
factorial study designs 





This Web site provides tools for assessing a study 
or project implementation through the RE-AIM lens. 
[Also relevant for: implementation and 
dissemination phase] 





Winhusen TC, Wilder C, Wexelblatt SL, et al. 
Design considerations for point-of-care clinical 
trials comparing methadone and buprenorphine 
treatment for opioid dependence in pregnancy 
and for neonatal abstinence syndrome. 
Contemp Clin Trials 2014;39(1):158-65. PMID: 
25183042. 

Funded by: The authors reported no funding 
assistance for the preparation of this journal 
article. 


This article describes the study design for a point- 
of-care RCT to treat pregnant women with opioid 
dependence and their newborns. It includes the 
design and the rationale for using this design. 





Woertman W, de Hoop E, Moerbeek M, et al. 
Stepped wedge designs could reduce the 
required sample size in cluster randomized 
trials. J Clin Epidemiol. Jul 2013;66(7):752-758. 
PMID: 23523551. 

Funded by: The authors received no financial 
support for the research, authorship, and/or 
publication of this article. 





This article discusses a sample size formula for 
cluster randomized stepped wedge designs. 
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Citation 
Wyrick DL, Rulison KL, Fearnow-Kenney M, et 
al. Moving beyond the treatment package 
approach to developing behavioral 
interventions: addressing questions that arose 
during an application of the Multiphase 
Optimization Strategy (MOST). Trans! Behav 
Med 2014;4(3):252-9. PMID: 25264465. 
Funded by: This project was funded by the 
National Institute on Drug Abuse (NIDA): 
3R44DA023735. 


Summary 
This article reports on the first-ever iterative 
application of Multiphase Optimization Strategy to 
an online program to prevent substance use among 
college student athletes. 





Zurovac J, Moreno L, Crosson J, et al. Using 
multifactorial experiments For comparative 
effectiveness research in physician practices 
with electronic health records. EGEMS 
2013;1(3):1037. PMID: 25848579 

Funded by: The authors reported no funding 
assistance for the preparation of this article. 


This article recommends the use of multifactorial 
designs and electronic health record data to 
evaluate quality improvement efforts in physician 
practices. 





Zurovac J, Peikes D, Zutshi A, Brown R. 
Efficient Orthogonal Designs: Testing the 
Comparative Effectiveness of Alternative Ways 
of Implementing Patient-Centered Medical 
Home Models. Rockville, MD: Agency for 
Healthcare Research and Quality. February 
2013. AHRQ Publication No. 13-0024-EF. 
(PDF — 6.64 MB). 
Funded by: AHRQ Contract No. 
HHSA29020090001 91 Task Order 5. 
Resources Relevant to the Imple 
Brownson RC, Colditz GA, Proctor EK. 
Dissemination and Implementation Research in 
Health: Translating Science to Practice. AHRQ 
PBRN Resource Center Webinar. 2015. 
https://obrn.ahrg.gov/events/dissemination- 
and-implementation-research-health- 


translating-science-practice. 
Funded by: AHRQ Contract No. HHSA290- 


2010-00004I, Task Order No. 32006T 

Cody S, Asher A. Proposal 14: Smarter, 
Better, Faster: The Potential for Predictive 
Analytics and Rapid-Cycle Evaluation to 
Improve Program Development and Outcomes. 
In Kearney MS, Harris BH (Eds.). Policies to 
Address Poverty in America. Washington, DC: 
Brookings; 2014. 

Funded by: The authors reported no funding 
assistance for the preparation of this book. 


This brief describes a study that used an efficient 

orthogonal design, a variation on factorial design. 

It discusses the pros and cons of the study design 
for comparative effectiveness research. 


mentation and Dissemination Phase 
In this Webinar, the discuss how to evaluate the 
evidence base on effective interventions; which 
strategies will produce the greatest impact; how to 
design an appropriate study; and how to track a set 
of essential outcomes as outlined in their book, 
Dissemination and Implementation Research in 
Health: Translating Science to Practice. 





This book discusses opportunities for applying 
analytic techniques from the private sector to public 
sector work. As government agencies invest in data 
warehouses and business intelligence capabilities, 
it becomes feasible to employ analytic techniques 
used more commonly in the private sector. 
Predictive analytics and rapid-cycle evaluation are 
analytical approaches that are used to do more 
than describe the current status of programs: in 
both the public and private sectors, these 
approaches provide decision-makers with guidance 





on what to do next. 
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OTE LiCeya) 
Gold M, Helms D, Guterman S. Identifying, 
monitoring, and assessing promising 
innovations: using evaluation to support rapid- 
cycle change. Issue Brief (Commonw Fund) 
2011;12:1-12. PMID: 21682057. 
Funded by: This paper was developed as part 
of a project funded by The Commonwealth 
Fund to AcademyHealth, with a subcontract to 
Mathematica Policy Research. 
Khanamen D. Thinking fast and slow. 
Cambridge University Press: New York; 2012. 
Funded by: The author reported no funding 
assistance for the preparation of this book. 
Lehmann CU, Conner KG, Cox JM. Preventing 
provider errors: online total parenteral nutrition 
calculator. Pediatrics 2004;113(4):748-53. 
PMID: 15060223. 
Funded by: This work was supported by the 
Center for Innovation and Quality Patient Care 
at the Johns Hopkins Hospital. 
Lewis CC, Stanick CF, Martinez RG, et al. The 
Society for Implementation Research 
Collaboration Instrument Review Project: a 
methodology to promote rigorous evaluation. 
Implement Sci. 2015 Jan;10(1):2. PMID: 
25567126. 
Funded by: This manuscript was supported, in 
kind, through the National Institutes of Health 
R13 award entitled, “Development and 
Dissemination of Rigorous Methods for 
Training and Implementation of Evidence- 
Based Behavioral Health Treatments” granted 
to Pl: KA Comtois from 2010 to 2015. Dr. 
Bryan J. Weiner’s time on the project was 
supported by the following funding: NIH CTSA 
at UNC UL1TRO00083. 


NIATx. Sustainability Model. 

http://www. niatx.net/Content/ContentPage.aspx 
2NID=156. 

Funded by: The University of Wisconsin- 
Madison Center for Health Enhancement 
Systems Studies (CHESS). 





Summary 
This issue brief suggests rapid-cycle evaluation, 
among other strategies, as a means of identifying 
desirable payment model innovations more quickly 
for CMS's Center for Medicare and Medicaid 
Innovation (CMMI). 


This book reports on human judgment and 
decisionmaking using concepts from psychology. 


This article discusses how low-cost, pragmatic 
approaches using Internet technology in the design 
of medical information systems can reduce medical 
errors and might pose a viable option for the 
prevention of adverse drug events. 


This paper discusses a study to identify 
psychometrically strong instruments for the field of 
implementation science. The paper discusses the 
methodology used by the Society for 
Implementation Research Collaboration (SIRC), 
and a summary of the preliminary review of over 
420 instruments and their scores, as assessed by 
constructs delineated in the Consolidated 
Framework for Implementation Research and the 
Implementation Outcomes Framework . 


Gaps in the field of dissemination and 
implementation science are discussed. 


Ultimately, SIRC hopes to develop an online 
repository where instruments can be reviewed by 
researchers and stakeholders on an ongoing basis. 
This will lead to the identification of 
psychometrically strong instruments through which 
to apply evidence-based implementation strategies 
to support widespread delivery of evidence-based 
care. 

This is a link to the Sustainability Index, a tool that 
can be used to assess the likelihood of an 
intervention or practice improvement being 
sustained in a health care setting. 
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Citation 
Norman N, Bennett C, Cowart S, et al. Boot 
camp translation: a method for building a 
community of solution. J Am Board Fam Med 
2013 Nov;26(3):254-63. PMID: 23657693. 
Funded by: This project was supported by 
Cooperative Agreement Number 5 U48 
DP000054-02 from the Centers for Disease 
Control and Prevention, 
an implementation grant from the Caring for 
Colorado Foundation for Community AIR and 
the NIH/_NCATS Colorado CTSI Grant 
Number UL1 TRO00154. 


Summary 
This article discusses the use of Boot Camp 
Translation in community-based settings to 
translate topics including asthma diagnosis and 
management, hypertension, and the patient- 
centered medical home, among others. 





Pickering BW, Herasevich V, Ahmed A, et al. 
Novel representation of clinical information in 
the ICU: developing user interfaces which 
reduce information overload. Appl Clin Inform 
2010;1(2):116-31. PMCID: PMC3632280. 
Funded by: This study was funded in part from 
intramural funds obtained through the Mayo 
Clinic, Critical Care Independent 
Multidisciplinary Practice committee and 
Innovation Loan Program. 


This paper outlines the barriers to effective use of 
electronic medical records and describes the 
methodology, using a worked example of the 
output. AWARE (Ambient Warning and Response 
Evaluation) is a physician-led, electronic- 
environment enhancement program in an 
academic, tertiary care institution’s intensive care 
unit (ICU). The development process focused on 
reducing information overload, improving efficiency 
and eliminating medical error in the ICU. 





Srinivasan M. The innovator’s DNA and 
healthcare improvement. J Gen Intern Med 
2014;29(1):1-2. 

PMCID: PMC3889935. 

Funded by: The author received no financial 
support for the research, authorship, and/or 
publication of this article. 


This article highlights the ways in which 
implementation science researchers currently apply 
the five characteristics of individuals and teams 
who successfully innovate from Jeff Dyer’s book, 
The Innovator’s DNA. The article concludes in a 
call to action to continue to innovate and improve 
health care delivery. 





Stevens BU, Yamada J, Promislow §, et al. 
Implementation of multidimensional knowledge 
translation strategies to improve procedural 
pain in hospitalized children. Implement Sci 
2014;9(1):120. PMID: 25928349. 

Funded by: The Canadian Institutes of Health 
Research (CIHR) (CTP-79854 and MOP- 
86605. 


Wen KY, Gustafson DH, Hawkins RP, et al. 
Developing and validating a model to predict 
the success of an IHCS implementation: the 
Readiness for Implementation Model. J Am 
Med Inform Assoc 2010;17(6):707—13. PMID: 
20962135. 

Funded by: This research was supported by 
the Agency for Healthcare Research and 
Quality (AHRQ RO1 HS10246). 





This article discusses a multifaceted knowledge 
translation (KT) intervention, Evidence-based 
Practice for Improving Quality (EPIQ), which 
included tailored KT strategies. This intervention 
was effective in improving pain practices and 
clinical outcomes at the unit level. The research 
was a prospective comparative cohort study in 32 
hospital units (16 EPIQ intervention and 16 
Standard Care), in eight pediatric hospitals in 
Canada. 

This article demonstrates the use of the Readiness 
for Implementation Model, which predicts a health 
care organization's potential for success in 
implementing an interactive health communication 
system. 





General Resources 
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OTE LiCey a) 
Auerbach AD, Landefeld CS, Shojania KG. The 
tension between needing to improve care and 
knowing how to do it. N Engl J Med. 
2007;357(6):608-13. PMID: 17687138. 
Funded by: AHRQ Contract No. (K08 
HS11416-02, to Dr. Auerbach), an Academic 
Leadership Award from the National Institute 
on Aging (KO7 AGO0912, to Dr. Landefeld), and 
funding from the Government of Canada 
Research Program (to Dr. Shojania). 


Summary 
This article argues that interventions to improve the 
quality and safety of health care should meet the 
same standards that are applied to the adoption of 
all medical technologies. 





The article discusses the pros and cons of rapid 
dissemination, and offers support for continuing to 
conduct randomized controlled trials as a tool for 
quality improvement. 





Centers for Medicare & Medicaid Services: 
2014 Physician Quality Reporting System 
(PQRS): Implementation Guide. Version 8.5. 
2014.1-43. 


http://www.cms.gov/apps/ama/license.asp?file= 
/PQRS/downloads/2014_ PQRS MeasuresList 


ImplementationGuide _12132013.zip. 
Funded by: The Centers for Medicare & 
Medicaid Services Physician Quality Reporting 
System. 


This document promotes understanding of how to 
implement 2013 PQRS claims-based reporting of 
measures in clinical practice, and to facilitate 
satisfactory reporting of quality data by eligible 
professionals who wish to participate in PQRS. 





Centers for Medicare & Medicaid Services: 
2014 Physician Quality Reporting System 
(PQRS) Measures List. Version 8.1.2014.1-54. 


http://www.cms.gov/apps/ama/license.asp?file= 
/PQRS/downloads/2014_ PQRS MeasuresList 


ImplementationGuide_12132013.zip. 
Funded by: The Centers for Medicare & 
Medicaid Services Physician Quality Reporting 
System. 


This document outlines the 2013 PQRS list of 
measures. This list is intended as a summary list to 
assist eligible professionals in initially reviewing the 
measures. 





deGruy F, Ewigman B, DeVoe J, et al. A plan 
for useful and timely family medicine and 
primary care research. Fam Med 2015;(In 
press). 

Funded by: Funding source unknown. 


This manuscript outlines a vision for the next 5 to 
10 years for family medicine research, including 
advocacy for placing "family medicine research" 
broadly in the context of primary care and 
population health research. 





Dyer JH, Gregersen HB, Christensen CM. The 
innovators DNA. Harvard Business Review 
2009; 87(12):60-67. PMID: 19968057. 
Funded by: The authors reported no funding 
assistance for the preparation of this article. 


This article outlines five “discovery skills” that allow 
the most creative business executives to maximize 
innovation production at their companies. The five 
discovery skills are: (1) associating, (2) 
questioning, (3) observing, (4) experimenting, and 
(5) networking. 





Green LA, Werner JJ, Etz RS. Contextual 
Relevancy and Research Collaborations, 
PBRNs Foster Partnerships for Pragmatic, 
Prompt Resolutions. AHRQ PBRN Resource 
Center Webinar. 
http://pbrn.ahrg.gov/events/contextual- 
relevancy-and-research-collaborations-pbrns- 


foster-partnerships-praqmatic-prompt. 
Funded by: AHRQ Contract No. HHSA290- 


2010-000041, Task Order No. 32006T 





This Webinar discusses the importance of 
expanding the breadth of PBRN research; 
redesigning research so it is more easily adoptable; 
and developing a primary care data model for 
PBRNs, with suggestions for how to do so. 
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OTE LiCey a) 
Gustafson DH, Hundt AS. Findings of 
innovation research applied to quality 
management principles for healthcare. Health 
Care Manage Rev 1995;20(2):16-33. PMID: 
7607882. 
Funded by: The Agency for Healthcare 
Research and Quality under the name of 
Agency for Health Care Policy and Research 
(AHCPR). 


Summary 
This study examined the difference between 
successful and failed change efforts to identify five 
characteristics that differentiated success from 
failure. 








Kairys SR, Wasserman R, Pace W. Practice- 
based quality improvement/research networks: 
full speed forward. Academic Pediatrics 2013 
Nov-Dec;13(6, Supplement):S12-S13. PMID: 
24268077. 

Funded by: AHRQ Contract No. 
1R01HS019908-01 


This commentary discusses the intersect of 
research and QI within the PBRN framework. The 
authors conclude that “The future may show us that 
networks can do it all: perform research, 
disseminate the results, and translate that research 
as well as QI efforts into sustainable improvements 
in health outcomes’. 





Maher L, Gustafson D, Evans A. Sustainability 
model and guide. NHS Institute for Innovation 
and Improvement. PDF. British National Health 
Service (NHS) Institute for Innovation and 
Improvement. 2007. 

http://www. institute.nhs.uk/index.php?option=c 
om joomcart&ltemid=194&main page=docum 
ent product info&cPath=67&products id=290 


&Joomcartid=2o0fvOdookq7c4dfpirss3dmvi5. 
Funded by: The British National Health 


Service. 


This is a detailed guide to the sustainability model, 
including a diagnostic score system and how to use 
it, designed to increase the likelinood of 
sustainability for health care improvements. 





Mathematica Policy Research. Rapid-Cycle 
Evaluation. Web site. http:/Awww.mathematica- 
mpr.com/our-capabilities/Rapid-Cycle- 
Evaluation. 

Funded by: The Brookings Institution. 


This Web site serves as a landing page for 
Mathematica’s work around “rapid-cycle 
evaluation,” including the aforementioned Smarter, 
Better, Faster White Paper. 





Mold JW, Peterson KA. Primary care practice- 
based research networks: working at the 
interface between research and quality 
improvement. Ann Fam Med 2005 May- 
Jun;3(suppl 1):S12-S20. PMID: 15928213. 
Funded by: The authors reported no funding 
assistance for the preparation of this journal 
article. 


Discusses the concept and examples of how QI 
processes are used to conduct research. 

[Also relevant for: the intersect of quality 
improvement (QI) and research] 





Patel DI, Puga F. Easy Approaches to Convert 
Quality Improvement to Research. 
Improvement Science Research Network 
Webinar. Webinar. 
http://www.isrn.net/Oct2012webseminar. 
Funded by: Funded in part by a grant from the 
Dean’s Scholarly Project Award Program: 
Scholarship of Teaching Award from the 
University of Texas Health Science Center San 
Antonio School of Nursing. 





This Webinar outlines processes and tools to build 
the capacity for collaboration and ensure 
successful work in team-based improvement 
research. The Webinar offers best practices for 
research collaboration, and describes how team- 
based science can increase innovation and 
advance knowledge. 
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Citation 
Peek CJ, Glasgow RE, Stange KC, et al. The 5 
R’s: an emerging bold standard for conducting 
relevant research in a changing world. Ann 


Fam Med 2014;12(5):447-45. PMID: 25354409. 


Funded by: NIH contract no. 
HHSN261200800001E. 


Summary 

This article addresses the assertion, “If we want 
more evidence-based practice, we need more 
practice-based evidence.” The authors outline a 
detailed description of an emerging standard for 
research, the “5 Rs,” and how to use them. These 
“5 Rs” stem from a synthesis of recommendations 
for care delivery research. 
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This report discusses suggested improvements in 
chronic illness care. Both chart review and self- 
report data on care processes and clinical 
outcomes suggested improvement based on 
changes teams made in the collaborative. Many of 
the organizations evidencing the largest 
improvements were community health centers, 
which had the fewest resources and the most 
challenged populations. 
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Appendix C. List of Working Group Participants 
AHRQ PBRN Working Group on Rapid-Cycle Research Participants 



































Elizabeth Waddell, PhD Wisconsin Research and Education Network (WREN) 

Geri Dino, PhD West Virginia Practice-Based Research Network 
(WVPBRN) 

Karim Keshavjee, MD Canadian Primary Care Sentinel Surveillance Network 
(CPCSSN) 

Michael Parchman, MD, MPH __| Group Health Research Institute 

Paul Meissner, MSPH New York City Research Improvement Networking Group 
(NYC RING) 

Kim Kimminau, PhD American Academy of Family Physicians National 
Research Network (AAFP NRN) 

Gerald Pulver, PhD, MPH University of Colorado Anschutz Medical Campus 

Amanda Ross, BS Collaborative Ohio Inquiry Network (COIN) 

Kurt Stange, MD, PhD Case Western Reserve University and RAP PBRN 
Khatri Knushbu, BS Weitzman Institute Safety Net Practice Based Research 
Network (WINS PBRN) 

Robert Penfold, PhD Group Health Research Institute 
Deborah Johnson, MHA The Dartmouth Practice-Based Research Network (The 
Dartmouth CO-OP Project) 
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Appendix D. Nominal Group Technique 
Below is a detailed overview of the five steps involved in the Nominal Group Technique. 
Step 1: Preparation 


Before using the Nominal Group Process, the meeting facilitator must complete a set of 
sequential preparatory tasks that set the stage for a successful meeting: 


1. Design preparation 
a. Prepare the NGT question that clarifies the objective of the meeting and illustrates 
the desired responses in terms of the level of abstraction and scope. The facilitator 
will often pilot test the question before to the meeting. 
b. Print the question on worksheets for each participant. 
c. Select the desired voting method (e.g., ranking vs. rating). 
2. Room preparation 
a. Secure a room large enough to comfortably seat group participants. 
b. Stock the room with these supplies: flip charts, masking tape, markers, pens, and 
paper for each participant, and either 3" x 5" index cards or post-it notes. 
3. Meeting preparation 
a. Prepare a welcome statement that explains the purpose of the meeting, outlines 
individual roles, and describes how the output will be used. 
b. Conduct the meeting following the NGT process. 


Step 2: Silent idea generation 


Before starting, the facilitator should prepare and present, in writing and verbally, the 
question that the group will consider in their meeting. A well-thought-out question will 
generate a wealth of potential ideas. The facilitator will encourage participants to silently and 
independently write ideas in brief phrases. 


The benefits of silent generation include: 

1. Allows adequate time for thinking and reflection through recall. 

2. Promotes social facilitation (e.g., seeing others hard at work). 

3. Avoids interruptions; undue focus on one idea; and competition, status, and 
conformance pressures, or choosing prematurely between ideas. 

4. Promotes a problem-centered focus. 


Step 3: Round-robin recording of ideas 


In this step, the facilitator goes around the table and records one idea from each participant 
on the flip chart. The ideas should be recorded verbatim with little to no paraphrasing. 
However, the facilitator is allowed to ask questions to clarify the idea. The process 
continues until all ideas have been recorded. Participants who are out of ideas can pass. 


The benefits of the round-robin recording are that it: 

1. Promotes equal participation in the presentation of ideas. 

2. Increases problem-mindedness and the ability to deal with a large number of ideas. 
3. Separates the ideas from the person. 

4. Allows for the tolerance of conflicting ideas. 
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5. Encourages hitchhiking on ideas. 
6. Provides written records of the ideas. 


Step 4: Serial discussion of ideas 


This involves taking each idea, one at a time (serially) and discussing or clarifying the idea 
before the preliminary vote. The benefits of this step are that it: 


Avoids unduly focusing on any one idea or a subset of ideas. 
Allows for clarification and elimination of any misunderstanding. 
Outlines the arguments and disagreements over ideas. 
Records differences of opinion without undue augmentation. 


Step 5: Voting 


PFoONs 


During this stage, group participants begin to narrow the list of potential ideas. Building on 
the discussion of ideas, individual members will make an independent judgment about those 
ideas they consider most likely to represent the problem to be solved or the potential 
solution to address it. 


The two voting methods, typically used, are ranking and rating. 
1. Rating method 


When rating the ideas, each participant distributes a set number of points (e.g., 100) across 
the ideas, as seen in Exhibit 1.1.: 


Exhibit 1.1. Rating Method 



































Idea # Joe Sue Kelly Jim Total 
1 50 50 
2 40 30 70 
3 20 100 32 152 
4 0 
5 30 30 
6 20 30 50 
7 0 
8 0 
9 20 5 18 43 
10 5 5 

Total 100 100 100 100 400 























As seen in the table above, each of the four team members distributed their points across 
the ten ideas they generated during Step 2 (silent idea generation). Note that participants 
have the option of assigning all of their points to one idea if they believe strongly that it is 
truly the best (i.e., Sue). From the table, note that Idea 3 has the highest point total, and the 
team can end the NGT process at this point, and choose this option. 


In another variation of this method, participants assign colored dots to ideas, using the same 
process. 
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2. Ranking method 


When ranking items, each participant is asked to choose roughly half of the total number of 
ideas generated and to rank these from most important to least important. This process will 
emphasize fewer ideas. In preparation for recording the vote, the facilitator should list the 
number of each idea on a separate piece of paper. When the actual votes are recorded, 
she/he will record the rank assigned by each participant to the idea, as seen in Exhibit 1.2.: 


Exhibit 1.2. Ranking Method 





























1 1 5 6 
2 5 5 10 
3 4 5 4 13 
4 0 
5 3 4 1 8 
6 2 4 3 2 11 
7 0 
8 2 2 
9 3 1 2 3 9 
10 1 1 


























As seen in the table above, Idea 3 has the highest score. In many instances, the nominal 
group technique will end after this step. If greater accuracy is desired, and especially if the 
group has generated a large number of ideas, the group may choose to engage in additional 
rounds of voting and discussion. 
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Appendix E. Using Change Concepts 


The presentation slides below were included with permission from their author, Lloyd 
Provost. The slides provide an overview of how to use Change Concepts, as derived from 
the book, The Improvement Guide: a Practical Approach to Enhancing Organizational 
Performance by Langley, et al, 2009. 


Change Concept: a general notion or approach to change 
that has been found to be useful in developing specific 
ideas for changes that lead to improvement. 


Concept 


An opportunity to create 
a new connection 





val 
[ 
Thought 
‘ 
— 


The Improvement Guide, 
Chapter 6, p. 134 
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Eliminate Waste 

Eliminate things that are not used 
Eliminate multiple entry 
Reduce or eliminate overkill 
Reduce controls on the system 
Recycle or reuse 

Use substitution 

Reduce classifications 
Remove intermediaries 

Match the amount to the need 
0. Use Sampling 

1 Change targets or set points 


RAO@ONAAAWN> 


Improve Work Flow 
12. Synchronize 
13. Schedule into multiple processes 
14, — Minimize handoffs 
15; Move steps in the process close together 
16. Find and remove bottlenecks 
17. Us automation 
18. — Smooth workflow 
19. Do tasks in parallel 
20. Consider people as in the same system 
21. Use multiple processing units 
22. Adjust to peak demand 


Optimize Inventory 
23 Match inventory to predicted demand 
24 Use pull systems 
25 Reduce choice of features 
26 Reduce multiple brands of the same item 


Change the Work Environment 
27. Give people access to information 
28. Use Proper Measurements 
29 Take Care of basics 
30. Reduce de-motivating aspects of pay system 
31. Conduct training 
32. Implement cross-training 
33: Invest more resources in improvement 
34. Focus on core process and purpose 
35. Share risks 
36. Emphasize natural and logical consequences 
37. Develop alliances/cooperative relationships 


Enhance the Producer/customer relationship 


38. Listen to customers 

39. Coach customer to use product/service 
40. Focus on the outcome to a customer 
41. Use a coordinator 

42. Reach agreement on expectations 

43. Outsource for “Free” 


44. Optimize level of inspection 
45. Work with suppliers 


Manage Time 

46. Reduce setup or startup time 
47. Set up timing to use discounts 
48. Optimize maintenance 

49. Extend specialist's time 

50. Reduce wait time 


Manage Variation 


51 Standardization (Create a Formal Process) 
52 Stop tampering 

53. Develop operation definitions 

54 Improve predictions 

55. Develop contingency plans 

56. Sort product into grades 

57 Desensitize 

58. Exploit variation 


Design Systems to avoid mistakes 


59. Use reminders 

60 Use differentiation 
61 Use constraints 
62 Use affordances 


Focus on the product or service 


63, Mass customize 

64 Offer product/service anytime 

65. Offer product/service anyplace 

66. Emphasize intangibles 

67. Influence or take advantage of fashion trends 
68. Reduce the number of components 

69. Disguise defects or problems 

70. Differentiate product using quality dimensions 
71 Move steps in process closer together 

72. Manage variation, not tasks 


Reference: The Improvement Guide, Langley, Nolan, Nolan, Norman 
and Provost, p.359 
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Change Concept 


- #51 


Standardize 


Perform the same task in the same way. Performi 
the same task in a variety of ways results in broa 


variation in 


practice, a reduced ability to monitor 


outcomes, and wasted time, effort, and money. 


Examples: 
Inhibitors, 


4) to increase the percentage of eligible pa 
implemen orders for initiation of ACEI's 


¢) implemen 
printed discharge 


fs 


t standardized discharge teaching methods, including pre- 


instructions that allow for customization for an 


individual patient; and 
od) document NYSA class, EF% and euvolemic weight in discharge 


summary. 


tisnts disc. 


on ACE 


the 
hospital stay for all patients admitted with a diagnosis of CHF, unless 
contraindicated; 
&) to inerease the percentage of patients with daily 
includs daily weights in a CHF management protocol that includes 
standing orders; and 


weights recorded, 


6 29 9-_ 
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Conceptual, vague, strategic Make an improvement 


| 


Redesign the process of 
order fulfillment 


Move steps in the process 
closer together 


Move order receipt and 
warehouse closer together 


Specific ideas, actionable Move the printer that prints 
orders into the warehouse 


IG, p. 135 
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